CakePHP ran out of memory after using find('all') on a large amount of data [Solved]

This is Hase from the development team

my previous article, I wrote about how to find ('all') a large amount of data in CakePHP without causing memory errors. However, I
received various comments from within the company regarding the content of that article, so
this time I will write about the feedback on the previous article.

So, I would be grateful if you would view my previous article as the ramblings of a novice programmer

Current code

$data = $this->Model->find('all'); // Do something .........

Using query() is a no-no

From the previous article

It seems that retrieving data using query() is lighter on processing than retrieving it using find('all').
So, I modified the code to retrieve data using query() instead of find.

This is incorrect

The `query` method is meant for executing complex queries, and
using it requires sanitization and validation, so it's generally best not to use it carelessly.
fact, using `find` is easier and more effective in utilizing the framework's features, so
there was absolutely no need to use the `query` method in this case.

rather than using a for statement.
It's better to use a while statement to retrieve data while looping,

Last time, we used a for loop to split and retrieve the data

Previous code

// Get the number of data items $count = $this->Model->find('count'); // Retrieve 5000 items of data at a time $limit = 5000; // Number of loops ({number of data items ÷ number of items retrieved at one time} rounded up) $loop = ceil($count / $limit); for ($i = 0; $i < $loop; $i++){ // Offset $offset = $limit * $i; $data = $this->Model->query("select * from hogehoge as limit {$limit} offset {$offset};", $cachequeries = false); // Do some processing ......... }

There's nothing wrong with this method, but

Actually, I've discovered that using a while loop is a much better method.
With a while loop, you don't need to get the count or calculate the number of loops.
Also, use find('all') instead of query().

So I rewrote it using a while statement

Rewritten code

// Retrieve 5000 data items at a time $limit = 5000; $params = array('limit' => $limit, 'offset' => 0); while ($data = $this->Model->find('all', $params)){ // Do some processing $params['offset']+= $limit; }

This method is far superior to using a for loop because it avoids unnecessary processing.
I learned something new!

That's all