[AUTOCUT] Gradle Check Flaky Test Report for IndexShardTests

[opensearch-ci-bot]

Flaky Test Report for IndexShardTests

Noticed the IndexShardTests has some flaky, failing tests that failed during post-merge actions.

Details

Git Reference	Merged Pull Request	Build Details	Test Name
17e8935	13844	39312	org.opensearch.index.shard.IndexShardTests.testRefreshLevelRestoreShardFromRemoteStore
1bdf43d	13866	39441	org.opensearch.index.shard.IndexShardTests.testRefreshLevelRestoreShardFromRemoteStore
679ccac	14248	40881	org.opensearch.index.shard.IndexShardTests.testRefreshLevelRestoreShardFromRemoteStore
a8b1734	18087	57282	org.opensearch.index.shard.IndexShardTests.testRestoreSearchOnlyShardFromStore

The other pull requests, besides those involved in post-merge actions, that contain failing tests with the IndexShardTests class are:

• 15333
• 17881
• 17457
• 17371
• 18333
• 16393
• 15990
• 15353
• 15922
• 14040
• 14065
• 14086
• 14341
• 14371
• 14415
• 14684
• 15586
• 17532
• 17889

For more details on the failed tests refer to OpenSearch Gradle Check Metrics dashboard.

[sachinpkale]

Fixed with #14418

[linuxpi]

[Storage Triage - attendees 1 2 3 4 5 6 7 8]

@sachinpkale Looks like the issue was not fixed! Please feel free to raise a PR to address!

[ashking94]

Tried reproducing this test by running the test for 2k+ iterations, but the failure did not occur. I also could not find any stack trace since all the build links have expired.

[andrross]

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.index.shard.IndexShardTests.testRestoreSearchOnlyShardFromStore" -Dtests.seed=90940F9D9F354EDC -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ee-TG -Dtests.timezone=Asia/Ashgabat -Druntime.java=21

IndexShardTests > testRestoreSearchOnlyShardFromStore FAILED
    org.apache.lucene.index.CorruptIndexException: misplaced codec footer (file truncated?): length=0 but footerLength==16 (resource=metadata__9223372036854775746__9223372036854775804__9223372036854775802__9223372036854775805__-1039764442__9223370290998210694__2)
        at __randomizedtesting.SeedInfo.seed([90940F9D9F354EDC:C6CDC876E014858E]:0)
        at app//org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:616)
        at app//org.opensearch.common.io.VersionedCodecStreamWrapper.readStream(VersionedCodecStreamWrapper.java:64)
        at app//org.opensearch.index.store.RemoteSegmentStoreDirectory.readMetadataFile(RemoteSegmentStoreDirectory.java:258)
        at app//org.opensearch.index.store.RemoteSegmentStoreDirectory.readLatestMetadataFile(RemoteSegmentStoreDirectory.java:247)
        at app//org.opensearch.index.store.RemoteSegmentStoreDirectory.init(RemoteSegmentStoreDirectory.java:154)
        at app//org.opensearch.index.store.RemoteSegmentStoreDirectory.<init>(RemoteSegmentStoreDirectory.java:141)
        at app//org.opensearch.index.shard.IndexShardTestCase.createRemoteSegmentStoreDirectory(IndexShardTestCase.java:814)
        at app//org.opensearch.index.shard.IndexShardTestCase.createRemoteStore(IndexShardTestCase.java:802)
        at app//org.opensearch.index.shard.IndexShardTestCase.newShard(IndexShardTestCase.java:672)
        at app//org.opensearch.index.shard.IndexShardTestCase.newShard(IndexShardTestCase.java:528)
        at app//org.opensearch.index.shard.IndexShardTestCase.reinitShard(IndexShardTestCase.java:892)
        at app//org.opensearch.index.shard.IndexShardTestCase.reinitShard(IndexShardTestCase.java:863)
        at app//org.opensearch.index.shard.IndexShardTestCase.reinitShard(IndexShardTestCase.java:858)
        at app//org.opensearch.index.shard.IndexShardTests.testRestoreSearchOnlyShardFromStore(IndexShardTests.java:3056)

@mch2 @vinaykpud Is this perhaps related to the reader/writer separation work?