SVSim/SVSimServer
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

refactor(bootstrap): finalize seed migration; remove GlobalsImporter and prod-captures plumbing

Browse Source 
Final cleanup of the bootstrap-seed refactor (Task 10 of 10):

- Delete the GlobalsImporter no-op stub and its two remaining call sites
  (Program.cs and SVSimTestFactory.cs). All work has moved to per-domain
  importers since Task 9.
- Drop the --captures CLI flag and CapturesDir / shippedCaptures plumbing
  from Program.cs (BootstrapOptions, ParseArgs, PrintUsage). Bootstrap input
  is now cards.json + reference CSVs + per-table seed JSON; no more
  prod-captures directory.
- Shrink ImporterBase from 141 to 23 lines: LoadCapture, Serialize,
  Upsert<T,TKey>, GetInt/GetString/GetBool/GetLong/GetULong all had zero
  callers after the seed migration. Only ParseWireDateTime survives (still
  used by PaymentItemImporter and MyPageGlobalsImporter for prod-shaped
  timestamp strings).
- Remove the prod-captures Content Include glob from SVSim.Bootstrap.csproj
  and both prod-captures globs (production + test-fixture overlay) from
  SVSim.UnitTests.csproj. Test fixtures now overlay production seeds at
  Data/seeds/ via the Task 7 layout exclusively.

Build clean; 391/391 unit tests passing.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
gamer147
2026-05-26 16:06:46 -04:00
parent d14a0be2c8
commit c02991a5c2
6 changed files with 26 additions and 186 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
SVSim.Bootstrap/Importers/GlobalsImporter.cs
View File

@@ -1,22 +0,0 @@
using SVSim.Database;
namespace SVSim.Bootstrap.Importers;
/// <summary>
/// Stub remaining after Stage 9C: the entire load-index → DB pipeline has been replaced by
/// per-domain importers in this folder (RotationConfigImporter, MyRotationImporter,
/// AvatarAbilityImporter, ArenaSeasonImporter, BattlePassImporter, DailyLoginBonusImporter,
/// PreReleaseInfoImporter, CardListsImporter, RotationFlagUpdater). Task 10 will delete this
/// class entirely; until then this stub keeps existing call sites compiling.
/// </summary>
public class GlobalsImporter
{
public Task<int> ImportAllAsync(SVSimDbContext context, string capturesDir)
{
// All work migrated to per-domain importers wired in Program.cs and
// SVSimTestFactory.SeedGlobalsAsync. Intentionally a no-op.
_ = context;
_ = capturesDir;
return Task.FromResult(0);
}
}

125
SVSim.Bootstrap/Importers/ImporterBase.cs
View File

@@ -1,84 +1,13 @@
using System.Text.Json;
namespace SVSim.Bootstrap.Importers;
/// <summary>
/// Shared helpers for content importers. Loads a prod-capture JSON file by endpoint name from
/// a captures directory, returning the inner <c>data</c> element. Picks the latest matching dated
/// file (e.g. <c>load-index-2026-05-23.json</c>) if multiple exist for the same endpoint.
/// Tiny shared helper for content importers. Capture parsing has moved out of the bootstrap
/// project entirely (extractors under <c>data_dumps/extract/</c> emit per-table seed JSON);
/// only the wire-date normaliser stays here because several seed-driven importers still need
/// to canonicalise prod-shaped timestamp strings.
/// </summary>
public static class ImporterBase
{
/// <summary>
/// Returns the parsed <c>.data</c> JsonElement for the latest <c>{endpoint}-*.json</c> file in
/// <paramref name="capturesDir"/>, or null if no file matches. Logs a warning when missing —
/// caller decides whether absence is fatal.
/// </summary>
public static JsonElement? LoadCapture(string capturesDir, string endpoint)
{
if (!Directory.Exists(capturesDir))
{
Console.Error.WriteLine($"[ImporterBase] Captures dir missing: {capturesDir}");
return null;
}
string? path = Directory.EnumerateFiles(capturesDir, $"{endpoint}-*.json")
.OrderByDescending(p => p)
.FirstOrDefault();
if (path is null)
{
Console.Error.WriteLine($"[ImporterBase] No capture found for endpoint '{endpoint}' in {capturesDir}");
return null;
}
using var fs = File.OpenRead(path);
using var doc = JsonDocument.Parse(fs);
if (!doc.RootElement.TryGetProperty("data", out var data))
{
Console.Error.WriteLine($"[ImporterBase] Capture file {path} has no top-level 'data' property.");
return null;
}
// Clone so the JsonElement survives doc disposal.
return data.Clone();
}
/// <summary>
/// Generic upsert by primary key. Returns (created, updated, unchanged) counts.
/// <paramref name="incoming"/> is the desired state from the capture; rows are matched by
/// <paramref name="keySelector"/>. <paramref name="applyChanges"/> mutates an existing row to
/// reflect incoming values and returns true if anything actually changed.
/// </summary>
public static (int created, int updated, int unchanged) Upsert<T, TKey>(
IEnumerable<T> incoming,
Dictionary<TKey, T> existingByKey,
Func<T, TKey> keySelector,
Action<T> addToContext,
Func<T, T, bool> applyChanges) where TKey : notnull
{
int created = 0, updated = 0, unchanged = 0;
foreach (var item in incoming)
{
var key = keySelector(item);
if (existingByKey.TryGetValue(key, out var existing))
{
if (applyChanges(existing, item)) updated++;
else unchanged++;
}
else
{
addToContext(item);
existingByKey[key] = item;
created++;
}
}
return (created, updated, unchanged);
}
/// <summary>Serialize a JsonElement back to compact JSON text for jsonb storage.</summary>
public static string Serialize(JsonElement el) =>
JsonSerializer.Serialize(el, new JsonSerializerOptions { WriteIndented = false });
/// <summary>Parse a wire date that may be ISO ("2026-05-23T..."), space-separated ("2026-05-23 16:32:31"), or empty.</summary>
public static DateTime ParseWireDateTime(string? s)
{
@@ -91,50 +20,4 @@ public static class ImporterBase
}
return DateTime.MinValue;
}
/// <summary>Read a JsonElement string/number property as long, defaulting on missing/null.</summary>
public static long GetLong(JsonElement el, string prop, long fallback = 0)
{
if (!el.TryGetProperty(prop, out var v) || v.ValueKind == JsonValueKind.Null) return fallback;
if (v.ValueKind == JsonValueKind.Number) return v.GetInt64();
if (v.ValueKind == JsonValueKind.String && long.TryParse(v.GetString(), out var n)) return n;
return fallback;
}
public static int GetInt(JsonElement el, string prop, int fallback = 0)
{
if (!el.TryGetProperty(prop, out var v) || v.ValueKind == JsonValueKind.Null) return fallback;
if (v.ValueKind == JsonValueKind.Number) return v.GetInt32();
if (v.ValueKind == JsonValueKind.String && int.TryParse(v.GetString(), out var n)) return n;
return fallback;
}
public static string GetString(JsonElement el, string prop, string fallback = "")
{
if (!el.TryGetProperty(prop, out var v) || v.ValueKind == JsonValueKind.Null) return fallback;
return v.ValueKind == JsonValueKind.String ? v.GetString() ?? fallback : v.ToString();
}
public static bool GetBool(JsonElement el, string prop, bool fallback = false)
{
if (!el.TryGetProperty(prop, out var v) || v.ValueKind == JsonValueKind.Null) return fallback;
if (v.ValueKind == JsonValueKind.True) return true;
if (v.ValueKind == JsonValueKind.False) return false;
if (v.ValueKind == JsonValueKind.Number) return v.GetInt32() != 0;
if (v.ValueKind == JsonValueKind.String)
{
var s = v.GetString();
if (bool.TryParse(s, out var b)) return b;
if (int.TryParse(s, out var i)) return i != 0;
}
return fallback;
}
public static ulong GetULong(JsonElement el, string prop, ulong fallback = 0)
{
if (!el.TryGetProperty(prop, out var v) || v.ValueKind == JsonValueKind.Null) return fallback;
if (v.ValueKind == JsonValueKind.Number) return v.GetUInt64();
if (v.ValueKind == JsonValueKind.String && ulong.TryParse(v.GetString(), out var n)) return n;
return fallback;
}
}

27
SVSim.Bootstrap/Program.cs
View File

@@ -34,7 +34,7 @@ public static class Program
Console.WriteLine($"[Bootstrap] Connection: {RedactPassword(opts.ConnectionString)}");
Console.WriteLine($"[Bootstrap] Reference CSVs: {opts.ReferenceDataDir}");
Console.WriteLine($"[Bootstrap] Cards file: {opts.CardsFile}");
Console.WriteLine($"[Bootstrap] Captures: {opts.CapturesDir}");
Console.WriteLine($"[Bootstrap] Seeds: {opts.SeedDir}");
var dbOptions = new DbContextOptionsBuilder<SVSimDbContext>()
.UseNpgsql(opts.ConnectionString)
@@ -75,9 +75,11 @@ public static class Program
if (!opts.SkipGlobals)
{
await new GlobalsImporter().ImportAllAsync(context, opts.CapturesDir);
// Load-index seed pipeline (Stage 9C replaced the old in-GlobalsImporter capture-parsing).
// Per-domain seed pipeline. The legacy GlobalsImporter that parsed prod-captured
// /load/index, /mypage/index, /deck/info wire payloads directly is gone — capture
// → seed transformation lives in data_dumps/extract/*; importers below just
// deserialise the per-table JSON files in SVSim.Bootstrap/Data/seeds/.
//
// RotationConfigImporter writes the Rotation GameConfig section that RotationFlagUpdater
// reads; CardImporter ran earlier in the !SkipCards block so CardSets are populated.
await new RotationConfigImporter().ImportAsync(context, opts.SeedDir);
@@ -136,7 +138,6 @@ public static class Program
private static BootstrapOptions? ParseArgs(string[] args)
{
string? cards = null;
string? captures = null;
string? referenceDataDir = null;
string? connection = null;
bool skipReference = false;
@@ -152,7 +153,6 @@ public static class Program
switch (a)
{
case "--cards": cards = NextArg(args, ref i); break;
case "--captures": captures = NextArg(args, ref i); break;
case "--reference-data-dir": referenceDataDir = NextArg(args, ref i); break;
case "--connection-string": connection = NextArg(args, ref i); break;
case "--skip-reference": skipReference = true; break;
@@ -170,15 +170,13 @@ public static class Program
}
// All bootstrap inputs ship in-project under SVSim.Bootstrap/Data/, copied next to the
// binary on build. The --cards/--captures/--reference-data-dir flags are ad-hoc overrides
// binary on build. The --cards/--reference-data-dir flags are ad-hoc overrides
// (e.g. point at a fresh loader dump before promoting it into the project).
string baseDir = AppContext.BaseDirectory;
string shippedDataDir = Path.Combine(baseDir, "Data");
string shippedCaptures = Path.Combine(shippedDataDir, "prod-captures");
string shippedCardsFile = Path.Combine(shippedDataDir, "cards.json");
string cardsFile = cards ?? positionalCards ?? shippedCardsFile;
string capturesDir = captures ?? shippedCaptures;
string refDir = referenceDataDir ?? shippedDataDir;
string shippedStoryDir = Path.Combine(shippedDataDir, "story");
string storyDir = storyDataDir ?? shippedStoryDir;
@@ -189,7 +187,7 @@ public static class Program
?? DefaultConnectionString;
return new BootstrapOptions(
cardsFile, capturesDir, refDir, connStr, skipReference, skipCards, skipGlobals,
cardsFile, refDir, connStr, skipReference, skipCards, skipGlobals,
skipStory, storyDir, shippedSeedDir);
}
@@ -212,23 +210,24 @@ public static class Program
" loader dump) — promote into Data/ when you're ready to make it permanent.\n" +
"\n" +
" --cards <file> Override path to cards.json (default: shipped Data/cards.json)\n" +
" --captures <dir> Override path to prod-captures directory\n" +
" (default: shipped Data/prod-captures)\n" +
" --reference-data-dir <dir> Override reference CSV directory (default: shipped Data/)\n" +
" --connection-string <conn> Postgres connection (or NPGSQL_CONNECTION env var,\n" +
$" then \"{DefaultConnectionString}\")\n" +
" --skip-reference Skip reference-data import (classes, sleeves, ranks, ...)\n" +
" --skip-cards Skip card + card-cosmetic-reward import\n" +
" --skip-globals Skip prod-captured globals import\n" +
" --skip-globals Skip seed-driven globals import (per-table JSON under Data/seeds)\n" +
" --story-data-dir <dir> Override story data directory (default: shipped Data/story)\n" +
" --skip-story Skip story import (worlds/sections/chapters/sbs)\n" +
"\n" +
"Capture-derived seeds are produced by extractors under data_dumps/extract/* and\n" +
"checked into SVSim.Bootstrap/Data/seeds/. The bootstrap project never parses wire\n" +
"captures directly — refresh seeds by re-running the relevant extractor.\n" +
"\n" +
"Back-compat: `svsim-bootstrap <cards.json> [connection]` still works (positional).");
}
private sealed record BootstrapOptions(
string CardsFile,
string CapturesDir,
string ReferenceDataDir,
string ConnectionString,
bool SkipReference,

3
SVSim.Bootstrap/SVSim.Bootstrap.csproj
View File

@@ -10,9 +10,6 @@
</PropertyGroup>
<ItemGroup>
<Content Include="Data\prod-captures\*.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
<Content Include="Data\*.csv">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>

17
SVSim.UnitTests/Infrastructure/SVSimTestFactory.cs
View File

@@ -176,21 +176,19 @@ internal sealed class SVSimTestFactory : WebApplicationFactory<Program>
}
/// <summary>
/// Runs <see cref="GlobalsImporter"/> against the test SQLite DB using the prod captures
/// copied into the test output dir (see SVSim.UnitTests.csproj Content Include for
/// Data/prod-captures). Idempotent — safe to call multiple times per factory. Tests that
/// depend on prod-shaped global content (spot_cards, avatar abilities, etc.) call this once
/// during setup; the rest of the test runs against whatever the importer populated.
/// Runs the per-domain seed importers against the test SQLite DB using the seed JSON
/// copied into the test output dir (see SVSim.UnitTests.csproj Content Includes for
/// Data/seeds and Data/test-fixtures/seeds). Idempotent — safe to call multiple times.
/// Tests that depend on prod-shaped global content (spot_cards, avatar abilities, etc.)
/// call this once during setup; the rest of the test runs against whatever the importers
/// populated. Mirrors the wiring in <see cref="SVSim.Bootstrap.Program"/>.
/// </summary>
public async Task SeedGlobalsAsync(string? capturesDir = null)
public async Task SeedGlobalsAsync()
{
capturesDir ??= Path.Combine(AppContext.BaseDirectory, "Data", "prod-captures");
string seedDir = Path.Combine(AppContext.BaseDirectory, "Data", "seeds");
using var scope = Services.CreateScope();
var ctx = scope.ServiceProvider.GetRequiredService<SVSimDbContext>();
await new GlobalsImporter().ImportAllAsync(ctx, capturesDir);
// Load-index seed pipeline (Stage 9C). Mirrors the wiring in SVSim.Bootstrap.Program.cs:
// RotationConfigImporter must precede RotationFlagUpdater; CardListsImporter is
// ordered after the GameConfig importers for tidiness (no FK dependency).
await new RotationConfigImporter().ImportAsync(ctx, seedDir);
@@ -203,7 +201,6 @@ internal sealed class SVSimTestFactory : WebApplicationFactory<Program>
await new CardListsImporter().ImportAsync(ctx, seedDir);
await new RotationFlagUpdater().UpdateAsync(ctx);
// Per-importer seed pipeline for the rest of the load-index split.
await new PracticeOpponentImporter().ImportAsync(ctx, seedDir);
await new PaymentItemImporter().ImportAsync(ctx, seedDir);
var puzzleImporter = new PuzzleImporter();

18
SVSim.UnitTests/SVSim.UnitTests.csproj
View File

@@ -36,14 +36,8 @@
<Content Include="..\SVSim.Bootstrap\Data\*.csv" Link="Data\%(Filename)%(Extension)">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
<!-- Tests that call SVSimTestFactory.SeedGlobalsAsync() need the prod captures available in
the test output dir. Mirror them from SVSim.Bootstrap so the same files drive both
production bootstrap and test seeding. -->
<Content Include="..\SVSim.Bootstrap\Data\prod-captures\*.json" Link="Data\prod-captures\%(Filename)%(Extension)">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
<!-- Seed JSON files for the per-importer pipeline (replaces prod-captures during migration).
Both globs coexist while individual endpoints are still being converted. -->
<!-- Production per-table seed JSON for the per-domain importer pipeline. Same files
consumed by SVSim.Bootstrap; mirrored here so SeedGlobalsAsync sees them. -->
<Content Include="..\SVSim.Bootstrap\Data\seeds\**\*.json" Link="Data\seeds\%(RecursiveDir)%(Filename)%(Extension)">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
@@ -54,14 +48,6 @@
<Content Include="..\SVSim.Bootstrap\Data\test-fixtures\seeds\*.json" Link="Data\seeds\%(Filename)%(Extension)">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
<!-- Test-only fixtures live outside prod-captures so the production bootstrap glob doesn't
pick them up (a fixture-named file would win the importer's reverse-alphabetical sort
against a dated capture). Linked into the same test output dir so SeedGlobalsAsync sees
them; per-feature tests rely on the fixture, the prod-capture smoke test routes around it
via a temp dir. -->
<Content Include="..\SVSim.Bootstrap\Data\test-fixtures\*.json" Link="Data\prod-captures\%(Filename)%(Extension)">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
<!-- StoryImporter integration test fixtures — copied to Story\Fixtures\ in test output. -->
<Content Include="Story\Fixtures\*.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>