Between Aug 6, ~22:05 UTC and Aug 7, 16:30 UTC, all Grain domains were unreachable. This included tag.grainql.com, clientapis.grainql.com, and grainql.com. Sites that embed the Grain script loaded slowly, up to 30 seconds in some cases. Grain did not collect analytics events in this period.
Cause
An error in our domain registration caused our DNS provider to change. The new servers accepted connections but did not respond. Browsers waited for the full connection timeout, which made customer pages slow.
Resolution
We corrected the domain registration on Aug 7. We restored the correct name servers and DNS records. DNS caches held the incorrect records for up to 12 hours, so recovery was gradual.
Follow-up actions
We added monitoring and safeguards for our domain registration.
We will add external uptime monitoring for tag.grainql.com and clientapis.grainql.com.
We will update the embed instructions so that a script outage cannot block customer pages.
Resolved
Between Aug 6, ~22:05 UTC and Aug 7, 16:30 UTC, all Grain domains were unreachable. This included tag.grainql.com, clientapis.grainql.com, and grainql.com. Sites that embed the Grain script loaded slowly, up to 30 seconds in some cases. Grain did not collect analytics events in this period.
Cause
An error in our domain registration caused our DNS provider to change. The new servers accepted connections but did not respond. Browsers waited for the full connection timeout, which made customer pages slow.
Resolution
We corrected the domain registration on Aug 7. We restored the correct name servers and DNS records. DNS caches held the incorrect records for up to 12 hours, so recovery was gradual.
Follow-up actions
We added monitoring and safeguards for our domain registration.
We will add external uptime monitoring for tag.grainql.com and clientapis.grainql.com.
We will update the embed instructions so that a script outage cannot block customer pages.
Investigating
Between Aug 6, ~22:05 UTC and Aug 7, 16:30 UTC, all Grain domains were unreachable. This included tag.grainql.com, clientapis.grainql.com, and grainql.com. Sites that embed the Grain script loaded slowly, up to 30 seconds in some cases. Grain did not collect analytics events in this period.
Cause
An error in our domain registration caused our DNS provider to change. The new servers accepted connections but did not respond. Browsers waited for the full connection timeout, which made customer pages slow.
Resolution
We corrected the domain registration on Aug 7. We restored the correct name servers and DNS records. DNS caches held the incorrect records for up to 12 hours, so recovery was gradual.
Follow-up actions
We added monitoring and safeguards for our domain registration.
We will add external uptime monitoring for tag.grainql.com and clientapis.grainql.com.
We will update the embed instructions so that a script outage cannot block customer pages.
