Apply BSA diffs to the database #2229
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| import org.junit.jupiter.api.Test; | ||
|
|
||
| /** Unit tests for {@link NonBlockedDomain}. */ | ||
| class NonBlockedDomainTest { |
Check notice
Code scanning / CodeQL
Unused classes and interfaces Note
| ImmutableMap<BlockList, String> fetchedChecksums = | ||
| ImmutableMap.of(BLOCK, block.peekChecksum(), BLOCK_PLUS, blockPlus.peekChecksum()); |
Check notice
Code scanning / CodeQL
Unread local variable Note
| ImmutableMap<BlockList, String> prevChecksums = | ||
| schedule | ||
| .latestCompleted() | ||
| .map(DownloadSchedule.CompletedJob::checksums) | ||
| .orElseGet(ImmutableMap::of); |
Check notice
Code scanning / CodeQL
Unread local variable Note
| checkArgument(columns.size() == 2, "Invalid line: [%s]", line); | ||
| checkArgument(!Strings.isNullOrEmpty(columns.get(0)), "Missing label in line: [%s]", line); | ||
| ImmutableList<Long> orderIds = | ||
| ORDER_SPLITTER.splitToStream(columns.get(1)).map(Long::valueOf).collect(toImmutableList()); |
Check notice
Code scanning / CodeQL
Missing catch of NumberFormatException Note
| import org.junit.jupiter.api.Test; | ||
|
|
||
| /** Unit tests for {@link Label}. */ | ||
| class LabelTest { |
Check notice
Code scanning / CodeQL
Unused classes and interfaces Note
| } | ||
|
|
||
| static ImmutableList<NonBlockedDomain> tallyNonBlockedDomainsForNewLabels( | ||
| ImmutableList<Label> labels, IdnChecker idnChecker, DownloadSchedule schedule, DateTime now) { |
Check notice
Code scanning / CodeQL
Useless parameter Note
weiminyu
force-pushed
the
bsa-5-apply-diff
branch
from
9be8b41
to
a76171a
Compare
| import org.testcontainers.shaded.com.google.common.collect.ImmutableList; | ||
|
|
||
| /** Unit tests for {@link JsonSerializations}. */ | ||
| class JsonSerializationsTest { |
Check notice
Code scanning / CodeQL
Unused classes and interfaces Note
core/src/main/java/google/registry/bsa/api/BsaReportSender.java
Dismissed
Show dismissed
Hide dismissed
weiminyu
force-pushed
the
bsa-5-apply-diff
branch
4 times, most recently
from
42eb420
to
161b7d1
Compare
| ImmutableList<Long> orderIds = | ||
| ORDER_SPLITTER | ||
| .splitToStream(columns.get(1)) | ||
| .map(Long::valueOf) |
Check notice
Code scanning / CodeQL
Missing catch of NumberFormatException Note
weiminyu
force-pushed
the
bsa-5-apply-diff
branch
from
161b7d1
to
f9e652f
Compare
Add the BsaDomainRefresh class which tracks the refresh actions. The refresh actions checks for changes in the set of registered and reserved domains, which are called unblockables to BSA.
Compare the latest BSA download and the previous one to discover: * New and deleted orders * New and deleted labels as well as konwn labels referenced by new orders The diff is stored on GCS as two files, one for orders and one for diffs.
Add a step in BsaDownloadAction to apply the diffs from the most recent download. This includes: - Add new labels and remove deleted labels to the `BsaLabel` table. - Add new entries to the `BsaDomainInUse` table if applicable. These are registered/reserved domains that match the new labels. - Generate list of domains that are not blocked and save them in a GCS file. These include all domains from the previous step as well as domains derived from invalid labels. More tests are needed but this PR successfully processed a real download.
weiminyu
force-pushed
the
bsa-5-apply-diff
branch
from
f9e652f
to
e2eb621
Compare
Add a step in BsaDownloadAction to apply the diffs from the most recent
download. This includes:
BsaLabeltable.BsaDomainInUsetable if applicable. These areregistered/reserved domains that match the new labels.
file. These include all domains from the previous step as well as
domains derived from invalid labels.
More tests are needed but this PR successfully processed a real
download.
This change is