This problem we had never observed before in any of the boards we had got assembled earlier. It was a very peculiar problem as if someone has put a time bomb inside the boards. :D
How did we come across this problem ?
Initially we got an order for test run of a Passenger bus vehicle tracking device and we shipped our VALTRACK-V3 (3G version) GPS tracker device for testing and it had won the confidence of the the fleet management company, so they had requested a second batch of shipment for deployment.
We got some boards assembled and shipped a second batch of VALTRACK-V3 boards which were fully tested in our R & D facility. This was a pilot run and not a big order. The devices functioned great in lab environments where the client tested the devices in their office before deployment. But after the devices were installed in the vehicle, they ran for a couple of days and then stopped sending data.
The devices were shipped back for testing and then we spent considerable amount of time to find the problem. Initially the device was not connecting to network, then we tried to touch up some pins of the SIM connector and antenna to see if the connections are proper, and to our surprise the devices started working. We thought the problem was fixed and then were almost ready to ship the devices back to the client. Before shipping the devices we keep them turned ON overnight for long term testing.
But all of the devices had stopped working in the middle of the night and paused sending data to the server. We were in deep confusion, as we touched the pads again with solder gun and they started working again !
We were puzzled as the earlier device which we had sent had the same hardware and the firmware in it as the new devices, Even the components used were from the same stock.
This made us go crazy, We found that the chips on the board were fine after replacing one by one. They functioned well in other boards. We came to the conclusion that, It's got to be the board itself which had issues. Somehow we did notice that the pads on the non working boards are not that shiny compared to the working boards. It was clear that we had soldering problem on this board. It was the classic Dry solder joint problem.
What's a Dry solder joint or Cold solder joint ?
When the solder is not able to create proper contact between the components pins and pads of the PCB a dry solder joint is created. This creates bad connectivity between the PCB pad and the parts pin. The dry joint is caused when there is oxide formation on the PCB pads which is quite common if you leave them exposed to open air. This oxide layer prevents the solder from flowing through the pads full contact area and prevents creating strong solder joints. Initially you might observe continuity between pads and component pin, but this can be due to mechanical stress and overtime when the joint becomes cold the metal contracts and makes the pad and pin disconnect causing the circuit to fail.
Dry joints can also occur if your Pad being heated has a large contact area and is dissipating heat faster than the solder gun can heat it, Even though the lead looks like it has melted on the surface it still might not have made proper contact to the pad leaving blur looking contacts.
How can we avoid a Dry solder joint?
The best solution is the solder flux which is used to wet the pads and the pins before soldering. The flux removes any oxide layer present and aids in creating strong solder joints by letting the solder flow through the entire pad area freely.
I have tried different fluxes but when you have the Dry joint problem i would suggest to use the Rosin type flux, This flux has yellowish tint and is not liked by many, but this is the best flux which has helped in fixing the dry joint problem. The small Quickfix solder flux paste you used in college time is a type of Rosin flux and is available for cheap at local shops in a circular tin box. You get liquid rosin flux too which is better as it can be applied easily. I don't recommend the Water based NC liquid fluxes as they were not able to fix the dry joint problem in our boards. Even though they make the PCB look dirty, the Rosin fluxes are better.
If you are soldering larger surfaces like ground plane pins then use a higher temperature/wattage soldering iron to heat the pad properly.
Its not a myth when they say about solder joints : If it ain't shiny, It ain't good. :)