Missing Int

10.7 Missing Int: Given an input file with four billion non-negative integers, provide an algorithm to generate an integer that is not contained in the file. Assume you have 1 GB of memory available for this task. FOLLOW UP What if you have only 10MB of memory? Assume that all the values are distinct and we now have no more than one billion non-negative integers.

• Brainstorm:

• Assuming 4 byte (32 bit) unsigned integers that represent 2^32 possible things. [0 - 2^32]

• 4 billion integers have to consume 128,000,000,000 bits or 16 Billion bytes which is roughly = 16x10^12/1x10^9 = 16 GB of memory. Memory space is insufficient to represent all numbers individually.

• Idea: Perform external merge sort (above). Build 16 hash tables, that each contain n/2 space (500MB) that each store the contents of the digits. I chose to do this as a preprocessing step. Then I would read in the hash tables one by one from disk, and iterate to find the first missing positive in O(N) time.

• Solution:

• Use a bit vector to that hashes the value of the unsigned integer into the bitset.

• 1 GB of memory can hold 2^30 bytes or 2^30 x 8 bits (8 billion) distinguishable things. Because we only have 4 billion elements, these can all hash to the bit vector that contains 4 billion boolean positions. Duplicates can be handled as well because we only need to flip the bit if it is true.

• Then, iterate through the array, and find the first missing index.

• This will take O(N) space, and O(N) time.

template<size_t size>
unsigned long int getFirstMissingNumber(bitset<size> bits) {
for (int i = 0; i < bits.size(); i++) {
if (!bits[i]) return i;
}
}

int main() {
srand(time(nullptr));
unsigned long int integers = 4000000000;

// (pow(2, 30) * 8) - 1 = 1GB = 8 billion bits
// 4294967295 (2^32-1) = unsigned long
bitset<ULONG_MAX> bits;
bits.reset(); // set to zero

for (unsigned long int i = 1; i <= integers + 1; i++) {
unsigned long int r = rand() % ULONG_MAX;
bits.set(r - 1, true);
}

cout << getFirstMissingNumber(bits) << endl;
}