Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create diffs between BSA downloads #2220

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

weiminyu
Copy link
Collaborator

@weiminyu weiminyu commented Nov 13, 2023

Compare the latest BSA download and the previous one to discover:

  • New and deleted orders

  • New and deleted labels as well as konwn labels referenced by new orders

The diff is stored on GCS as two files, one for orders and one for diffs.


This change is Reviewable

import org.junit.jupiter.api.Test;

/** Unit tests for {@link NonBlockedDomain}. */
class NonBlockedDomainTest {

Check notice

Code scanning / CodeQL

Unused classes and interfaces Note

Unused class: NonBlockedDomainTest is not referenced within this codebase. If not used as an external API it should be removed.
core/src/test/java/google/registry/bsa/api/OrderTest.java Dismissed Show dismissed Hide dismissed
Comment on lines +97 to +98
ImmutableMap<BlockList, String> fetchedChecksums =
ImmutableMap.of(BLOCK, block.peekChecksum(), BLOCK_PLUS, blockPlus.peekChecksum());

Check notice

Code scanning / CodeQL

Unread local variable Note

Variable 'ImmutableMap<BlockList,String> fetchedChecksums' is never read.
Comment on lines +99 to +103
ImmutableMap<BlockList, String> prevChecksums =
schedule
.latestCompleted()
.map(DownloadSchedule.CompletedJob::checksums)
.orElseGet(ImmutableMap::of);

Check notice

Code scanning / CodeQL

Unread local variable Note

Variable 'ImmutableMap<BlockList,String> prevChecksums' is never read.
import org.junit.jupiter.api.Test;

/** Unit tests for {@link Label}. */
class LabelTest {

Check notice

Code scanning / CodeQL

Unused classes and interfaces Note

Unused class: LabelTest is not referenced within this codebase. If not used as an external API it should be removed.
@weiminyu weiminyu added the WIP Work in progress. Don't review yet. label Nov 14, 2023
@weiminyu weiminyu force-pushed the bsa-4-make-diff branch 5 times, most recently from 941c8e4 to 92de21e Compare November 17, 2023 22:01
@weiminyu weiminyu force-pushed the bsa-4-make-diff branch 2 times, most recently from 3b11d56 to 73de4c5 Compare November 29, 2023 16:06
Copy link
Collaborator Author

@weiminyu weiminyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 46 files reviewed, 7 unresolved discussions (waiting on @Github-advanced-security[bot])


core/src/main/java/google/registry/bsa/BsaDownloadAction.java line 98 at r1 (raw file):

Previously, github-advanced-security[bot] wrote…

Unread local variable

Variable 'ImmutableMap<BlockList,String> fetchedChecksums' is never read.

Show more details

Will be used when BSA checksum is available


core/src/main/java/google/registry/bsa/BsaDownloadAction.java line 103 at r1 (raw file):

Previously, github-advanced-security[bot] wrote…

Unread local variable

Variable 'ImmutableMap<BlockList,String> prevChecksums' is never read.

Show more details

Will be used when BSA checksum is available


core/src/main/java/google/registry/bsa/api/Order.java line 42 at r1 (raw file):

Previously, github-advanced-security[bot] wrote…

Missing catch of NumberFormatException

Potential uncaught 'java.lang.NumberFormatException'.

Show more details

Fixed in next pr


core/src/test/java/google/registry/bsa/api/LabelTest.java line 25 at r1 (raw file):

Previously, github-advanced-security[bot] wrote…

Unused classes and interfaces

Unused class: LabelTest is not referenced within this codebase. If not used as an external API it should be removed.

Show more details

spurious github check


core/src/test/java/google/registry/bsa/api/NonBlockedDomainTest.java line 24 at r1 (raw file):

Previously, github-advanced-security[bot] wrote…

Unused classes and interfaces

Unused class: NonBlockedDomainTest is not referenced within this codebase. If not used as an external API it should be removed.

Show more details

spurious github check


core/src/test/java/google/registry/bsa/api/OrderTest.java line 24 at r1 (raw file):

Previously, github-advanced-security[bot] wrote…

Unused classes and interfaces

Unused class: OrderTest is not referenced within this codebase. If not used as an external API it should be removed.

Show more details

spurious github check

Copy link
Collaborator Author

@weiminyu weiminyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 46 files reviewed, 7 unresolved discussions (waiting on @Github-advanced-security[bot])


core/src/main/java/google/registry/bsa/BsaDiffCreator.java line 162 at r1 (raw file):

Previously, github-advanced-security[bot] wrote…

Missing catch of NumberFormatException

Potential uncaught 'java.lang.NumberFormatException'.

Show more details

Done.

ImmutableList<Long> orderIds =
ORDER_SPLITTER
.splitToStream(columns.get(1))
.map(Long::valueOf)

Check notice

Code scanning / CodeQL

Missing catch of NumberFormatException Note

Potential uncaught 'java.lang.NumberFormatException'.
Copy link
Collaborator Author

@weiminyu weiminyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 46 files reviewed, 8 unresolved discussions (waiting on @Github-advanced-security[bot])


core/src/main/java/google/registry/bsa/BsaDiffCreator.java line 165 at r4 (raw file):

Previously, github-advanced-security[bot] wrote…

Missing catch of NumberFormatException

Potential uncaught 'java.lang.NumberFormatException'.

Show more details

Fixed in the next pr

Add the BsaDomainRefresh class which tracks the refresh actions.

The refresh actions checks for changes in the set of registered and
reserved domains, which are called unblockables to BSA.
Compare the latest BSA download and the previous one to discover:

* New and deleted orders

* New and deleted labels as well as konwn labels referenced by new orders

The diff is stored on GCS as two files, one for orders and one for
diffs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
WIP Work in progress. Don't review yet.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant