[Laravel] What does "Maximum retries of 10000 reached without finding a unique value" mean? [Faker]

Hello,
I'm Mandai, the Wild Team member of the development team.

I think most people will use
the bundled Faker It's convenient because it's multilingual and allows you to create a variety of test data, but it also has a convenient method called unique that allows you to create multiple pieces of data with unique IDs, which is also very useful.

However, depending on how you use it, you may get an error like the one in the title, and you may not know what it means, or you may not be able to resolve it and have to try a different method... If that happens, it may seem like a waste of having installed Faker! So, this time I would like to introduce how to resolve the error "Maximum retries of 10,000 reached without finding a unique value" that is often seen with Faker

 

Cause of the error

Judging from the error message, I have a feeling that I've gotten stuck in an infinite loop somewhere, but I can't think of anything specific, and all I know is that the error occurs somewhere around the time I use Faker's unique(),
since I haven't written any code that would cause an infinite loop.

If you reset it at the right time, it works fine, but of course, when you reset it, duplication occurs.
You won't know how to get the hang of this until you've been using it for a while, but once you understand the cause, you should be able to figure out how to deal with it...!!

So, let's take a look at the code for Faker's unique() method

 

Tracking unique() behavior

This part of github is the code for the unique() method.
The received variables are passed directly to a class called UniqueGenerator, so let's take a closer look at the code for the UniqueGenerator class.

The behavior is well controlled using magic methods, but essentially it just calls a method on the Faker\Generator object received when the object was created with UniqueGenerator::__call().
The while condition checks for duplicates, so the returned value is unique.
Checking for duplicates using array keys seems quite primitive, but the cause is easy to understand.
The error message when an OverflowException is thrown includes the title text, and the cause was that the number of loops in this do while statement had reached the retry limit.

 

How to avoid errors

There is only one adjustment you can make to avoid the error: change the value of the second argument of the unique() method to increase the number of retries the loop.
The default is 10,000, but if the number of values ​​you want is 15,000, it will definitely overflow.

To increase the number of retries:

$factory->define(User::class, function(Faker\Generator $faker) { return [ 'name' => $faker->name(), 'age' => $faker->unique(false, 15000)->numberBetween(1, 80), // Change the number of retries to 15000 ]; });

 

Another cause is that the number of variations in the values ​​returned by the Faker\Generator object is less than 10,000 to begin with.
As in the example above, numberBetween() can only return values ​​from 1 to 80, so a unique value cannot be generated for the 81st piece of data, and the loop continues beyond the retry limit.
It may seem like there is a wide range of variation in random combinations such as people's names, but while there are a fair number of variations in English first names (male and female) - around 3,000 and last names (473), in Japanese there are only around 50 variations for first names (male and female), 31 for surnames, and only around 1,500 for full names.

Even if you want 5,000 full names in Japanese, if there are only 1,500 possible combinations, using unique() will be meaningless, so you'll need to think of another way.
For example, there seems to be a way to implement and add a new provider to the Faker\Generator class, but I haven't looked into it in detail yet, so I hope to be able to introduce it at some point.

 

summary

This time, I looked into the error "Maximum retries of 10000 reached without finding a unique value" that I often come across when using Faker.
It all started when I was researching the error message online and came across a post on StackOverflow that said "reset(true) is fine!", which made me think that something was wrong.

I wondered if there was some fundamental misunderstanding, so I looked into it, but it seems there was more than one pitfall

If you come across the message "Maximum retries of 10,000 reached without finding a unique value," investigate the number of retries and the amount of dummy data obtained

 
That's all