tag:storjdcs.statuspage.io,2005:/historyStorj DCS Status - Incident History2024-03-29T05:23:09ZStorj DCStag:storjdcs.statuspage.io,2005:Incident/199712372024-02-13T20:35:46Z2024-02-13T20:35:46ZIntermittent timeouts while connecting to edge services within proximity of Los Angeles Metropolitan Area<p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>20:35</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>02:18</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>01:14</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>01:09</var> UTC</small><br><strong>Investigating</strong> - We are investigating reports of performance impacts to one or more components of our system.</p>tag:storjdcs.statuspage.io,2005:Incident/197462832024-01-29T17:09:06Z2024-01-29T17:09:06ZInvestigating reports of slower uploads<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>17:09</var> UTC</small><br><strong>Resolved</strong> - While we are continuing to monitor this issue, p90 and p95 upload metric values are returning to our expected ranges. We anticipate some upcoming changes will continue to positively affect these values in the coming weeks. For the interim, we have scaled backend resources accordingly to account for any spikes in load.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>17:13</var> UTC</small><br><strong>Update</strong> - Issues have been identified, and fixes have been scheduled. Updating the the state to monitoring until the changes can be applied.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>17:09</var> UTC</small><br><strong>Monitoring</strong> - Issues have been identified, and fixes have been scheduled. Updating the the state to monitoring until the changes can be applied.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>05:13</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>17:13</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>02:36</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:storjdcs.statuspage.io,2005:Incident/198213582024-01-24T20:28:59Z2024-01-24T20:29:00ZIncreased satellite error rates especially with listing objects<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>20:28</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>20:11</var> UTC</small><br><strong>Monitoring</strong> - The fix has been implemented and deployed and we are monitoring the results.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>19:57</var> UTC</small><br><strong>Update</strong> - We are continuing to work on this issue. Satellite operations have degraded performance.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>19:44</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being deployed.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>19:38</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:storjdcs.statuspage.io,2005:Incident/196066962024-01-04T00:43:06Z2024-01-04T00:43:06ZSmall percentage of gateway requests with slow response rates<p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>00:43</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>00:28</var> UTC</small><br><strong>Update</strong> - Performance has returned to normal, we are continuing to monitor for any issues.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>00:10</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>23:51</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>23:41</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating the issue</p>tag:storjdcs.statuspage.io,2005:Incident/194381292023-12-14T10:35:00Z2023-12-14T13:21:10ZConnectivity Outages to Edge Services in Eastern North America Between 10:35 UTC – 10:48 UTC<p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>10:35</var> UTC</small><br><strong>Resolved</strong> - Customers in eastern part of North America might have experienced a series of intermittent connectivity issues where clients would indicate that no network route is available to edge services within the 10:35 UTC – 10:48 UTC window. We apologize for any inconvenience caused, and we thank you for your patience.</p>tag:storjdcs.statuspage.io,2005:Incident/194313892023-12-13T20:00:00Z2023-12-13T20:43:01ZConnectivity Outages to Edge Services in Asia, North America and South America Between 20:00 UTC – 20:19 UTC<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>20:00</var> UTC</small><br><strong>Resolved</strong> - Customers in Asia, North America and South America might have experienced a series of intermittent connectivity issues for up to a maximum of 1-minute within the 20:00 UTC – 20:19 UTC window. We apologize for any inconvenience caused, and thank you for your patience.</p>tag:storjdcs.statuspage.io,2005:Incident/193744562023-12-08T12:06:37Z2023-12-08T12:14:25ZIncreased Edge Services’ Latency in North America<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>12:06</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. We thank you for your patience while we worked on resolving the issue.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>11:36</var> UTC</small><br><strong>Monitoring</strong> - Users in the North American region should no longer be experiencing increased latency.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>09:08</var> UTC</small><br><strong>Identified</strong> - Due to a sudden load spike in the North American region, customers might be experiencing increased latency while they are load-balanced to other regions. The on-call engineering team is monitoring this incident closely and are scaling on-demand capacity. We apologize for any inconvenience caused.</p>tag:storjdcs.statuspage.io,2005:Incident/193659272023-12-07T12:16:11Z2023-12-07T12:16:11ZUnavailability of stats.storjshare.io<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>12:16</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. We thank you for your patience while we worked on resolving the issue.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>11:46</var> UTC</small><br><strong>Monitoring</strong> - The on-call team has finished applying the fix, and we are currently waiting for DNS to propagate.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>11:19</var> UTC</small><br><strong>Identified</strong> - Due to an invalid DNS configuration applied to stats.storjshare.io, it has become temporarily unavailable. The on-call team has identified the issue and is currently applying a fix. We apologize for any inconvenience caused.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>11:14</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:storjdcs.statuspage.io,2005:Incident/192436452023-11-29T17:31:09Z2023-11-29T17:31:09ZTemporary Unavailability of the Main Website for Users in the US Region<p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>17:31</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>17:21</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>17:20</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>17:17</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:storjdcs.statuspage.io,2005:Incident/192002282023-11-23T09:39:09Z2023-11-23T09:39:09ZIntermittent Timeouts for Users of Edge Services in Europe<p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>09:39</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>09:09</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>07:43</var> UTC</small><br><strong>Investigating</strong> - Customers using edge services in Europe might be experiencing problems connecting to S3 gateway and Link Sharing services. We are currently investigating this issue.</p>tag:storjdcs.statuspage.io,2005:Incident/189704292023-10-30T17:15:00Z2023-10-30T18:07:00Z(Europe) S3 Gateway Responding Slowly to TLS Handshakes<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>17:15</var> UTC</small><br><strong>Resolved</strong> - European customers might have experienced degraded performance, connection errors and intermittent timeouts due to slow TLS handshakes while interfacing with Storj S3 gateway.<br /><br />The issue occurred because of misconfigured limits of our edge capacity in Europe – our limits being too restrictive in the number of connections, causing connections to queue up and be processed slowly.<br /><br />We have adjusted our limits and are monitoring the results.<br /><br />We apologize to all who were affected by the disruption.</p>tag:storjdcs.statuspage.io,2005:Incident/189154242023-10-24T21:50:02Z2023-10-24T21:50:02ZStorj.io getting 'Too Many Requests' in US<p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>21:50</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>21:49</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>17:59</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and storj.io is no longer returning 'too many requests'. We are continuing to monitor.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>17:58</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>17:48</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating the issue</p>tag:storjdcs.statuspage.io,2005:Incident/188142292023-10-16T20:45:50Z2023-10-16T20:45:50ZEmail sending issue<p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>20:45</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>13:43</var> UTC</small><br><strong>Identified</strong> - We have identified an issue that is affecting our ability to send emails (password reset, project invite, signup emails). We are actively working on a fix for this issue.</p>tag:storjdcs.statuspage.io,2005:Incident/187379652023-10-10T02:47:11Z2023-10-10T02:47:11ZPacket Loss to Edge Services Affecting Customers Globally<p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>02:47</var> UTC</small><br><strong>Resolved</strong> - The issue has been resolved for all affected users.<br /><br />We thank you for your patience while we worked on resolving the issue.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>01:28</var> UTC</small><br><strong>Monitoring</strong> - Description: Mitigation work is completed by our engineering team.<br /><br />We apologize to all who are affected by the disruption.<br /><br />Diagnosis: Customers might have experienced increased packet loss to edge services globally.</p>tag:storjdcs.statuspage.io,2005:Incident/183653152023-09-02T23:46:46Z2023-09-02T23:46:46ZUS1: Link Sharing Service Increased Errors and Latency<p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>23:46</var> UTC</small><br><strong>Resolved</strong> - The issue with Link Sharing Service has been resolved for all affected users.<br /><br />We thank you for your patience while we worked on resolving the issue.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>23:33</var> UTC</small><br><strong>Monitoring</strong> - Description: we have added extra capacity to the affected datacenter, and are monitoring the situation closely.<br /><br />Diagnosis: Link Sharing Service customers may continue to experience increased latency.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>23:18</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>22:54</var> UTC</small><br><strong>Identified</strong> - Description: Mitigation work is currently underway by our engineering team.<br /><br />We do not have an ETA for mitigation at this point.<br /><br />We apologize to all who are affected by the disruption.<br /><br />Diagnosis: Link Sharing Service customers will experience increased errors and latency.<br /><br />Workaround: None at this time.</p>tag:storjdcs.statuspage.io,2005:Incident/171424982023-05-03T14:00:22Z2023-05-04T11:32:27ZIncorrect API responses for objects not found<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>Resolved</strong> - A deployment on 5/2/2023 at 5 PM UTC introduced a regression that made edge services return an (HTTP) 500 Internal Server Error server error response code instead of correct responses for objects not found.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:storjdcs.statuspage.io,2005:Incident/165314652023-03-16T18:27:39Z2023-03-16T18:27:39ZIncreased load causing elevated error rates on US1<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>18:27</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>18:15</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues. Service levels have been restored.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>18:07</var> UTC</small><br><strong>Monitoring</strong> - The fix has been rolled out and error rates are stabilizing.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>17:51</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being deployed.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>17:35</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating the issue and will provide an update at 17:45 UTC.</p>tag:storjdcs.statuspage.io,2005:Incident/163190602023-03-06T23:24:23Z2023-03-06T23:24:23Zreduced S3 gateway performance in US (east region)<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>23:24</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>18:43</var> UTC</small><br><strong>Update</strong> - We are working on a fix, and are continuing to monitor the issue.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>23:52</var> UTC</small><br><strong>Monitoring</strong> - Team scaling services to meet demand. Some throttling in-place as we scale.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>Identified</strong> - Significantly increased demand impacting performance. Team scaling services to meet demand.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:storjdcs.statuspage.io,2005:Incident/158146322023-01-11T18:00:53Z2023-01-11T18:00:53ZIssues with fetching and displaying the linksharing map<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>16:52</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>22:30</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>22:10</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>21:10</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an elevated error rate on the linksharing service.</p>tag:storjdcs.statuspage.io,2005:Incident/127199482022-11-04T19:45:57Z2022-11-04T20:07:33ZDegraded Performance for Gateway/Linksharing<p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>19:45</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>17:28</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:45</var> UTC</small><br><strong>Identified</strong> - Intermittent unavailability of registration and retrieval of newly registered access grants.</p>tag:storjdcs.statuspage.io,2005:Incident/107678182022-08-05T22:59:55Z2022-08-05T22:59:55ZElevated error rates on eu1.storj.io<p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>22:59</var> UTC</small><br><strong>Resolved</strong> - A fix has been deployed and the incident is resolved.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:storjdcs.statuspage.io,2005:Incident/106935602022-07-27T01:40:17Z2022-07-27T01:40:17ZSTORJ tokens deposits aren't working<p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>01:40</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>01:40</var> UTC</small><br><strong>Update</strong> - We have a short-term secure workaround now in place, but we will be working to improve functionality ASAP.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>13:19</var> UTC</small><br><strong>Investigating</strong> - All the Satellite operations are working except depositing the STORJ tokens.<br />We are able to reproduce it and we are investigating the issue.</p>tag:storjdcs.statuspage.io,2005:Incident/103961122022-07-06T22:58:39Z2022-07-06T22:58:39ZCopy Object functionality service degredation<p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>22:58</var> UTC</small><br><strong>Resolved</strong> - Server-side object copy has been re-enabled on all satellites.</p><p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>19:48</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>18:49</var> UTC</small><br><strong>Update</strong> - We have increased the amount of teams working to re-enable server-side object copy. We will continue to work on this as our top priority until it is fixed.</p><p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>15:19</var> UTC</small><br><strong>Identified</strong> - Support for server-side object copy is currently disabled while we work to mitigate some recently uncovered scaling issues.</p>tag:storjdcs.statuspage.io,2005:Incident/102389662022-06-12T02:01:51Z2022-06-12T02:01:51ZEU1 High load, intermittent timeouts<p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>02:01</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>22:12</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>18:04</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:storjdcs.statuspage.io,2005:Incident/100813162022-05-26T21:58:07Z2022-05-26T21:58:07ZIntermittent login failures<p><small>May <var data-var='date'>26</var>, <var data-var='time'>21:58</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>19:22</var> UTC</small><br><strong>Monitoring</strong> - Mitigations are working as expected. We are continuing to monitor the situation.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>17:57</var> UTC</small><br><strong>Identified</strong> - We have identified the issue and have put mitigations in place.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>16:19</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>13:15</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue with intermittent login failures on the US1 satellite.</p>