MySQL - Optimize Query To Remove Using Temporary and Using filesort

Question

Query

SELECT SQL_NO_CACHE
 `Table1`.`recordID`
FROM
  `Table1`
  LEFT JOIN `Table3` ON `Table3`.`table1RecordID`=`Table1`.`recordID`
WHERE
  (`Table3`.`status` = '3' OR `Table3`.`status` = '4')  AND
  (`Table1`.`groupName` = 'Sample Name')
GROUP BY `Table3`.`recordID` ASC, `Table1`.`recordID` ASC;

Explain

+----+-------------+--------+--------+------------------------------+---------+---------+------------------------------+-------+----------------------------------------------+
| id | select_type | table  |  type  |        possible_keys         |   key   | key_len |             ref              | rows  |                    Extra                     |
+----+-------------+--------+--------+------------------------------+---------+---------+------------------------------+-------+----------------------------------------------+
|  1 | SIMPLE      | Table3 | ALL    | fk_packageID,regStatus,pkgID | NULL    | NULL    | NULL                         | 11322 | Using where; Using temporary; Using filesort |
|  1 | SIMPLE      | Table1 | eq_ref | PRIMARY,groupName            | PRIMARY | 4       | testDb.Table3.table1RecordID |     1 | Using where                                  |
+----+-------------+--------+--------+------------------------------+---------+---------+------------------------------+-------+----------------------------------------------+

---

If I remove the second part of the GROUP BY, "Table1.recordID ASC" but data isn't correct when I do that. Why is it doing this and how can I fix it and still group down by Table1 in addition to the Table3 first.

Thanks in advanced!

Update 1/24/14

I had time to take the full query and pull the tables to a generic form to post without client data. I was able to add schema to sqlfiddle but without the data I'm using results can be different and I was even unable to put 100 rows pre-table (7 total) into sqlfiddle due to limitations of characters. So instead I've done a dump of the tables and I'm sharing it over dropbox.

Dropbox

https://www.dropbox.com/s/9fgu626996utpar/stackoverflow-21291707_test_db_schema_and_data.sql

Query

SELECT
  `t1`.`name` AS `Object1.Name`,
  GROUP_CONCAT(DISTINCT
  IF(`t5`.`questionID`=68,
    IF(`t6`.`writeInRequired` = 1,
      CONCAT(
        `t6`.`value`,
        ':', `t5`.`writeInResponse`
      ),
      `t6`.`value`
    ),
    NULL
  ) SEPARATOR ', ') AS `Object3.Response_68`,
  GROUP_CONCAT(DISTINCT
  IF(`t5`.`questionID`=67,
    IF(`t6`.`writeInRequired` = 1,
      CONCAT(
        `t6`.`value`,
        ':', `t5`.`writeInResponse`
      ),
      `t6`.`value`
    ),
    NULL
  ) SEPARATOR ', ') AS `Object3.Response_67`,
  GROUP_CONCAT(DISTINCT
  IF(`t5`.`questionID`=66,
    IF(`t6`.`writeInRequired` = 1,
      CONCAT(
        `t6`.`value`,
        ':', `t5`.`writeInResponse`
      ),
      `t6`.`value`
    ),
    NULL
  ) SEPARATOR ', ') AS `Object3.Response_66`,
  `t7`.`firstName` AS `Object8.FirstName`,
  `t7`.`lastName` AS `Object8.LastName`,
  `t7`.`email` AS `Object8.Email`,
  `t1`.`recordID` AS `Object1.PackageID`,
  `t3`.`recordID` AS `Object5.RegistrationID`
FROM
  `Table1` t1
  LEFT JOIN `Table2` t2 ON `t1`.`recordID`=`t2`.`table1RecordID`
  LEFT JOIN `Table3` t3 ON `t3`.`table1RecordID`=`t1`.`recordID`
  LEFT JOIN `Table4` t4 ON `t4`.`table3RecordID`=`t3`.`recordID` AND `t4`.`type` = 1
  LEFT JOIN `Table5` t5 ON `t5`.`objectID`=`t3`.`recordID` AND `t5`.`objectType`='Type2'
  LEFT JOIN `Table6` t6 ON `t6`.`recordID`=`t5`.`table6RecordID`
  JOIN `Table7` t7 ON `t7`.`recordID`=`t4`.`table7RecordID`
WHERE
  `t3`.`status` IN ('3','4')
GROUP BY
  `Object5.RegistrationID` ASC,
  `Object1.PackageID` ASC

EXPLAIN EXTENDED (/G)

*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: t7
         type: ALL
possible_keys: PRIMARY
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 11627
     filtered: 100.00
        Extra: Using temporary; Using filesort
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: t4
         type: ref
possible_keys: idx_table7RecordID,idx_table3RecordID
          key: idx_table7RecordID
      key_len: 5
          ref: testDb.t7.recordID
         rows: 1
     filtered: 100.00
        Extra: Using where
*************************** 3. row ***************************
           id: 1
  select_type: SIMPLE
        table: t3
         type: eq_ref
possible_keys: PRIMARY,table1RecordID_status,idx_status,idx_table1RecordID
          key: PRIMARY
      key_len: 4
          ref: testDb.t4.table3RecordID
         rows: 1
     filtered: 100.00
        Extra: Using where
*************************** 4. row ***************************
           id: 1
  select_type: SIMPLE
        table: t5
         type: ref
possible_keys: compositeIDs
          key: compositeIDs
      key_len: 773
          ref: const,testDb.t3.recordID
         rows: 5
     filtered: 100.00
        Extra:
*************************** 5. row ***************************
           id: 1
  select_type: SIMPLE
        table: t6
         type: eq_ref
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 4
          ref: testDb.t5.table6RecordID
         rows: 1
     filtered: 100.00
        Extra:
*************************** 6. row ***************************
           id: 1
  select_type: SIMPLE
        table: t1
         type: eq_ref
possible_keys: PRIMARY
          key: PRIMARY
      key_len: 4
          ref: testDb.t3.table1RecordID
         rows: 1
     filtered: 100.00
        Extra:
*************************** 7. row ***************************
           id: 1
  select_type: SIMPLE
        table: t2
         type: ref
possible_keys: idx_table1RecordID
          key: idx_table1RecordID
      key_len: 5
          ref: testDb.t1.recordID
         rows: 85
     filtered: 100.00
        Extra: Using index
7 rows in set, 1 warning (0.13 sec)

EXPLAIN EXTENDED (ASCII TABLE)

+----+-------------+-------+--------+-------------------------------------------------------------+--------------------+---------+--------------------------+-------+----------+---------------------------------+
| id | select_type | table |  type  |                        possible_keys                        |        key         | key_len |           ref            | rows  | filtered |              Extra              |
+----+-------------+-------+--------+-------------------------------------------------------------+--------------------+---------+--------------------------+-------+----------+---------------------------------+
|  1 | SIMPLE      | t7    | ALL    | PRIMARY                                                     | NULL               | NULL    | NULL                     | 11627 | 100.00   | Using temporary; Using filesort |
|  1 | SIMPLE      | t4    | ref    | idx_table7RecordID,idx_table3RecordID                       | idx_table7RecordID | 5       | testDb.t7.recordID       |     1 | 100.00   | Using where                     |
|  1 | SIMPLE      | t3    | eq_ref | PRIMARY,table1RecordID_status,idx_status,idx_table1RecordID | PRIMARY            | 4       | testDb.t4.table3RecordID |     1 | 100.00   | Using where                     |
|  1 | SIMPLE      | t5    | ref    | compositeIDs                                                | compositeIDs       | 773     | const,testDb.t3.recordID |     5 | 100.00   |                                 |
|  1 | SIMPLE      | t6    | eq_ref | PRIMARY                                                     | PRIMARY            | 4       | testDb.t5.table6RecordID |     1 | 100.00   |                                 |
|  1 | SIMPLE      | t1    | eq_ref | PRIMARY                                                     | PRIMARY            | 4       | testDb.t3.table1RecordID |     1 | 100.00   |                                 |
|  1 | SIMPLE      | t2    | ref    | idx_table1RecordID                                          | idx_table1RecordID | 5       | testDb.t1.recordID       |    85 | 100.00   | Using index                     |
+----+-------------+-------+--------+-------------------------------------------------------------+--------------------+---------+--------------------------+-------+----------+---------------------------------+

Thanks again in advanced!

For @Strawberry

Removed select clauses and just select the recordID of the first table. This query still produces the same explain from above.

SELECT
  `t1`.`recordID`
FROM
  `Table1` t1
  LEFT JOIN `Table2` t2 ON `t1`.`recordID`=`t2`.`table1RecordID`
  LEFT JOIN `Table3` t3 ON `t3`.`table1RecordID`=`t1`.`recordID`
  LEFT JOIN `Table4` t4 ON `t4`.`table3RecordID`=`t3`.`recordID` AND `t4`.`type` = 1
  LEFT JOIN `Table5` t5 ON `t5`.`objectID`=`t3`.`recordID` AND `t5`.`objectType`='Type2'
  LEFT JOIN `Table6` t6 ON `t6`.`recordID`=`t5`.`table6RecordID`
  JOIN `Table7` t7 ON `t7`.`recordID`=`t4`.`table7RecordID`
WHERE
  `t3`.`status` IN ('3','4')
GROUP BY
  `t3`.`recordID` ASC,
  `t1`.`recordID` ASC;

Answer 1

Some comments about your query:

1. The Left outer join is unnecessary. You are undoing the outer join with the where clause.
2. There is no need to group by both fields, Table1.recordID is sufficient.
3. in is more readable than a bunch of comparisons with or.

Here is the resulting query:

SELECT SQL_NO_CACHE t1.`recordID`
FROM `Table1` t1 JOIN
     `Table3` t3
     ON t3.`table1RecordID` = t1.`recordID`
WHERE t3.`status` in ('3', '4')  AND
      t1.`groupName` = 'Sample Name'
GROUP BY t1.`recordID` ASC;

Does this perform better?

If not, let's think about what the query is doing. It is trying to filter out records from t1 based on things in t3. Assuming that table1.RecordId is unique, then this would work:

SELECT SQL_NO_CACHE t1.`recordID`
FROM `Table1` t1 
WHERE t1.`groupName` = 'Sample Name' and
      exists (select 1
              from Table3 t3
              where t3.`status` in ('3', '4') and
                    t1.RecordId = t3.RecordId
             );

For best performance, you should have an index on Table3(RecordId, status).