{"id":13460,"externalId":null,"message":{"pathToResolution":null,"actionPlan":null,"rootCause":null},"additionalInformation":"","isCore":true,"affectsAll":true,"status":"Resolved","type":"Disruption","createdAt":"2024-11-15T18:27:25.792Z","updatedAt":"2025-04-17T19:57:47.881Z","instanceKeys":["AP45","AP46","AP48","AP49","AP57","AP59","AP61","AP62","CS242","CS246","CS249","CS251","CS253","CS254","CS258","CS259","CS264","CS268","CS270","CS273","CS276","CS280","CS294","CS295","CS310","CS312","CS313","CS316","CS319","CS320","CS345","EU50","NA125","NA226","NA231","NA232","NA233","NA241","NA249","NA251"],"serviceKeys":["coreService"],"IncidentImpacts":[{"id":14851,"startTime":"2024-11-15T15:32:00.000Z","endTime":"2024-11-16T04:46:00.000Z","type":"serviceDisruption","severity":"major","createdAt":"2024-11-15T18:27:25.820Z","updatedAt":"2025-04-17T19:57:48.040Z","startTimeCreatedAt":"2024-11-15T18:27:25.821Z","startTimeModifiedAt":"2024-11-21T13:15:26.135Z","endTimeCreatedAt":"2024-11-16T05:15:26.768Z","endTimeModifiedAt":null}],"IncidentEvents":[{"id":11976,"type":"rootCauseAnalysis","message":"The root cause analysis for this incident has been updated to include detailed next step actions including associated delivery dates and additional feature-level information. The RCA is available here: https://help.salesforce.com/s/articleView?id=003876113&type=1","createdAt":"2024-12-12T23:46:01.100Z","updatedAt":"2025-04-17T19:57:48.020Z"},{"id":11897,"type":"rootCauseAnalysis","message":"An updated Root Cause Analysis (RCA) is now available here: https://help.salesforce.com/s/articleView?id=003876113&type=1. Further updates will be made when more information becomes available.\n\nNote that following our investigation, the Technology team discovered that a previously identified group of instances initially considered impacted were not affected, and as a result, these instances have been removed from the Root Cause Analysis (RCA)","createdAt":"2024-11-22T22:41:40.830Z","updatedAt":"2025-04-17T19:57:48.006Z"},{"id":11862,"type":"rootCauseAnalysis","message":"We sincerely apologize for the impact this issue caused you and your business. Our goal is to provide world-class service to our customers, and we are continuously assessing and improving our tools, processes, and architecture to provide our customers with the best service possible.  The preliminary root cause analysis is now available here: https://help.salesforce.com/s/articleView?id=003876113&type=1. This link will be updated with more information as it becomes available. If you have additional questions or need further support, please open a case with our Support team via the Help portal.","createdAt":"2024-11-17T21:59:39.717Z","updatedAt":"2025-04-17T19:57:47.964Z"},{"id":11860,"type":"update","message":"The Salesforce Technology team successfully implemented the fix, resolving the issue across all impacted instances. Following validation checks, the team confirmed full resolution and declared an all-clear at 04:46 UTC on November 16, 2024.\n\nWe’ll perform a full investigation of the incident, confirming the technical trigger, the underlying cause, and preventive action to avoid a recurrence.","createdAt":"2024-11-16T05:15:26.775Z","updatedAt":"2025-04-17T19:57:47.971Z"},{"id":11859,"type":"update","message":"The Salesforce Technology team is implementing a permissions fix on remaining instances and closely monitoring performance across all impacted instances to ensure stability and minimize any impact.\n\nWe’ll provide an update upon significant or sooner if additional information becomes available.","createdAt":"2024-11-16T03:08:57.857Z","updatedAt":"2025-04-17T19:57:47.977Z"},{"id":11858,"type":"impactRadiusIncrease","message":"Investigations have highlighted that the impact radius is broader than initially understood. We have updated the posting to reflect the additional instances experiencing impact, and customers in these instances will receive communications related to this incident going forward.  ","createdAt":"2024-11-16T02:57:22.471Z","updatedAt":"2025-04-17T19:57:47.983Z"},{"id":11857,"type":"update","message":"The Salesforce Technology team reviewed the Test results and identified permissions issues on the Test instance. These permissions were fixed, and initial reports indicate that services have returned to normal on the Test instance. However, out of an abundance of caution, the team is continuing to monitor performance levels and perform additional testing.  \n\nWe’ll provide another update upon significant update or sooner if additional information becomes available.","createdAt":"2024-11-16T02:25:22.517Z","updatedAt":"2025-04-17T19:57:47.990Z"},{"id":11856,"type":"update","message":"The Salesforce technology team continues to validate the test results from the test instance and monitor performance levels. \n\nWe’ll provide another update upon significant update or sooner if additional information becomes available.","createdAt":"2024-11-16T01:54:18.430Z","updatedAt":"2025-04-17T19:57:47.996Z"},{"id":11855,"type":"update","message":"The technology team is exploring a configuration issue as a potential trigger. We are conducting additional tests and validating results on the Test instance.\n\nWe’ll provide another update upon significant update or sooner if additional information becomes available.","createdAt":"2024-11-16T00:59:43.759Z","updatedAt":"2025-04-17T19:57:48.027Z"},{"id":11854,"type":"update","message":"The initial release to one instance did not yield positive results, and our teams continue to work toward identifying the trigger of this issue. As our investigation continues, we have also identified that the impact may be different from what was initially understood. A subset of customers have been experiencing a service disruption, and this post has been changed to reflect that. \n\nWe’ll provide another update upon significant update or sooner if additional information becomes available.","createdAt":"2024-11-15T22:48:09.718Z","updatedAt":"2025-04-17T19:57:47.928Z"},{"id":11853,"type":"update","message":"We have identified a potential solution and will be following a phased release process. The initial release is expected to take up to two hours. Once complete, we will monitor the environment for stability and proceed with a release to all impacted instances. \n\nWe’ll provide another update upon the completion of the test release or sooner if additional information becomes available.","createdAt":"2024-11-15T21:10:19.443Z","updatedAt":"2025-04-17T19:57:47.936Z"},{"id":11851,"type":"update","message":"We are continuing to investigate the trigger for this incident and are pursuing multiple paths forward while validating each approach in a test environment to mitigate the impact as quickly as possible. This issue has also impacted the NA231 instance. \n\nWe’ll provide an update in 60 minutes or sooner if additional information becomes available.","createdAt":"2024-11-15T20:31:04.826Z","updatedAt":"2025-04-17T19:57:47.943Z"},{"id":11849,"type":"update","message":"Our teams have made progress on narrowing the impact radius, and continue to work towards identifying the trigger. During this analysis, it was determined that the issue impacts instances CS259, CS280, and CS276. The analysis also indicates instances AP54 and NA248 are not impacted.\n\nWe’ll provide an update in 60 minutes or sooner if additional information becomes available.","createdAt":"2024-11-15T19:33:35.706Z","updatedAt":"2025-04-17T19:57:47.951Z"},{"id":11848,"type":"update","message":"At 15:32 UTC, on November 15, 2024, the salesforce technology team identified a feature degradation that is preventing users from accessing newly created records for up to 20 minutes. We are actively investigating the trigger of the issue. \n\nWe’ll provide an update in 60 minutes or sooner if additional information becomes available.","createdAt":"2024-11-15T18:27:25.821Z","updatedAt":"2025-04-17T19:57:47.957Z"}],"timeline":[{"title":"Update","content":"The root cause analysis for this incident has been updated to include detailed next step actions including associated delivery dates and additional feature-level information. The RCA is available here: https://help.salesforce.com/s/articleView?id=003876113&type=1","createdAt":"2024-12-12T23:46:01.100Z","updatedAt":"2025-04-17T19:57:48.020Z","entryType":"event","sourceId":11976,"sourceType":"event"},{"title":"Update","content":"An updated Root Cause Analysis (RCA) is now available here: https://help.salesforce.com/s/articleView?id=003876113&type=1. Further updates will be made when more information becomes available.\n\nNote that following our investigation, the Technology team discovered that a previously identified group of instances initially considered impacted were not affected, and as a result, these instances have been removed from the Root Cause Analysis (RCA)","createdAt":"2024-11-22T22:41:40.830Z","updatedAt":"2025-04-17T19:57:48.006Z","entryType":"event","sourceId":11897,"sourceType":"event"},{"title":"Update","content":"We sincerely apologize for the impact this issue caused you and your business. Our goal is to provide world-class service to our customers, and we are continuously assessing and improving our tools, processes, and architecture to provide our customers with the best service possible.  The preliminary root cause analysis is now available here: https://help.salesforce.com/s/articleView?id=003876113&type=1. This link will be updated with more information as it becomes available. If you have additional questions or need further support, please open a case with our Support team via the Help portal.","createdAt":"2024-11-17T21:59:39.717Z","updatedAt":"2025-04-17T19:57:47.964Z","entryType":"event","sourceId":11862,"sourceType":"event"},{"title":"Update","content":"The Salesforce Technology team successfully implemented the fix, resolving the issue across all impacted instances. Following validation checks, the team confirmed full resolution and declared an all-clear at 04:46 UTC on November 16, 2024.\n\nWe’ll perform a full investigation of the incident, confirming the technical trigger, the underlying cause, and preventive action to avoid a recurrence.","createdAt":"2024-11-16T05:15:26.775Z","updatedAt":"2025-04-17T19:57:47.971Z","entryType":"event","sourceId":11860,"sourceType":"event"},{"title":"Resolved","content":"Incident Resolved","createdAt":"2024-11-16T05:15:26.768Z","updatedAt":null,"entryType":"impact_end","endTime":"2024-11-16T04:46:00.000Z","sourceId":14851,"sourceType":"impact","impactStartTime":"2024-11-15T18:27:25.821Z"},{"title":"Update","content":"The Salesforce Technology team is implementing a permissions fix on remaining instances and closely monitoring performance across all impacted instances to ensure stability and minimize any impact.\n\nWe’ll provide an update upon significant or sooner if additional information becomes available.","createdAt":"2024-11-16T03:08:57.857Z","updatedAt":"2025-04-17T19:57:47.977Z","entryType":"event","sourceId":11859,"sourceType":"event"},{"title":"Update","content":"Investigations have highlighted that the impact radius is broader than initially understood. We have updated the posting to reflect the additional instances experiencing impact, and customers in these instances will receive communications related to this incident going forward.  ","createdAt":"2024-11-16T02:57:22.471Z","updatedAt":"2025-04-17T19:57:47.983Z","entryType":"event","sourceId":11858,"sourceType":"event"},{"title":"Update","content":"The Salesforce Technology team reviewed the Test results and identified permissions issues on the Test instance. These permissions were fixed, and initial reports indicate that services have returned to normal on the Test instance. However, out of an abundance of caution, the team is continuing to monitor performance levels and perform additional testing.  \n\nWe’ll provide another update upon significant update or sooner if additional information becomes available.","createdAt":"2024-11-16T02:25:22.517Z","updatedAt":"2025-04-17T19:57:47.990Z","entryType":"event","sourceId":11857,"sourceType":"event"},{"title":"Update","content":"The Salesforce technology team continues to validate the test results from the test instance and monitor performance levels. \n\nWe’ll provide another update upon significant update or sooner if additional information becomes available.","createdAt":"2024-11-16T01:54:18.430Z","updatedAt":"2025-04-17T19:57:47.996Z","entryType":"event","sourceId":11856,"sourceType":"event"},{"title":"Update","content":"The technology team is exploring a configuration issue as a potential trigger. We are conducting additional tests and validating results on the Test instance.\n\nWe’ll provide another update upon significant update or sooner if additional information becomes available.","createdAt":"2024-11-16T00:59:43.759Z","updatedAt":"2025-04-17T19:57:48.027Z","entryType":"event","sourceId":11855,"sourceType":"event"},{"title":"Update","content":"The initial release to one instance did not yield positive results, and our teams continue to work toward identifying the trigger of this issue. As our investigation continues, we have also identified that the impact may be different from what was initially understood. A subset of customers have been experiencing a service disruption, and this post has been changed to reflect that. \n\nWe’ll provide another update upon significant update or sooner if additional information becomes available.","createdAt":"2024-11-15T22:48:09.718Z","updatedAt":"2025-04-17T19:57:47.928Z","entryType":"event","sourceId":11854,"sourceType":"event"},{"title":"Update","content":"We have identified a potential solution and will be following a phased release process. The initial release is expected to take up to two hours. Once complete, we will monitor the environment for stability and proceed with a release to all impacted instances. \n\nWe’ll provide another update upon the completion of the test release or sooner if additional information becomes available.","createdAt":"2024-11-15T21:10:19.443Z","updatedAt":"2025-04-17T19:57:47.936Z","entryType":"event","sourceId":11853,"sourceType":"event"},{"title":"Update","content":"We are continuing to investigate the trigger for this incident and are pursuing multiple paths forward while validating each approach in a test environment to mitigate the impact as quickly as possible. This issue has also impacted the NA231 instance. \n\nWe’ll provide an update in 60 minutes or sooner if additional information becomes available.","createdAt":"2024-11-15T20:31:04.826Z","updatedAt":"2025-04-17T19:57:47.943Z","entryType":"event","sourceId":11851,"sourceType":"event"},{"title":"Update","content":"Our teams have made progress on narrowing the impact radius, and continue to work towards identifying the trigger. During this analysis, it was determined that the issue impacts instances CS259, CS280, and CS276. The analysis also indicates instances AP54 and NA248 are not impacted.\n\nWe’ll provide an update in 60 minutes or sooner if additional information becomes available.","createdAt":"2024-11-15T19:33:35.706Z","updatedAt":"2025-04-17T19:57:47.951Z","entryType":"event","sourceId":11849,"sourceType":"event"},{"title":"Disruption Started","content":"serviceDisruption","createdAt":"2024-11-15T18:27:25.821Z","updatedAt":"2024-11-21T13:15:26.135Z","entryType":"impact_start","startTime":"2024-11-15T15:32:00.000Z","sourceId":14851,"sourceType":"impact","impactStartTime":"2024-11-15T18:27:25.821Z"},{"title":"Update","content":"At 15:32 UTC, on November 15, 2024, the salesforce technology team identified a feature degradation that is preventing users from accessing newly created records for up to 20 minutes. We are actively investigating the trigger of the issue. \n\nWe’ll provide an update in 60 minutes or sooner if additional information becomes available.","createdAt":"2024-11-15T18:27:25.821Z","updatedAt":"2025-04-17T19:57:47.957Z","entryType":"event","sourceId":11848,"sourceType":"event"}]}