• Information Security graduate
• Geography enthusiast
• Photography lover
• Fan of 房东的猫 (Fangdong's Cat)

For me, blogging isn't about packaging everything neatly — it's about preserving "how I judged the situation at the time, what pitfalls I hit, and how it was ultimately resolved." Next time I encounter a similar problem, I can take fewer detours; when others find it through search, they can quickly judge whether this approach suits them.

AI

The AI category mainly records experiences with large language models and related tools.

Currently planned topics:

• Local LLM deployment, e.g., LM Studio, Ollama.
• Tool chains like DeepSeek, Cherry Studio, SiliconFlow.
• API calls, model integration, and some low-cost usage methods.
• Practical uses of AI tools in daily work.

Articles in this section will be practice-oriented, not just theory. It's only worth recording if it can be run locally, integrated with a client, and solve real problems.

AWS

AWS currently has the most content on the blog. This section mainly covers cloud service troubleshooting, EC2 operations, Windows/Linux system issues, S3, ALB, IAM, FSx, SSM, CloudWatch, and similar cases.

These articles typically follow this structure:

1. Problem symptoms
2. Key logs
3. Investigation path
4. Root cause diagnosis
5. Solution
6. Follow-up recommendations

Many AWS problems look like cloud platform issues, but the root cause may end up being in the operating system, certificate chain, KMS permissions, Windows Update, AD ports, Kerberos tickets, or third-party security software. I write these down as a reminder: troubleshoot in layers, don't attribute the problem to a specific component right from the start.

DevOps

The DevOps category covers more general server, container, and service migration records.

For example:

• 1Panel server maintenance
• Docker networking and iptables/nftables issues
• Migrating new-api from SQLite to MySQL
• Blue-green production service switching
• Reverse proxy and database connection issues

The emphasis in this type of content is reproducibility and rollback capability. Operations in production environments can't just aim for "it works" — you also need to consider backups, verification, traffic switching, and failure rollback.

VPS

The VPS category records servers, networking, domains, and common scripts.

Planned additions:

• Common Linux scripts
• VPS benchmarking and network testing
• Domain, DNS, and certificate configuration
• Nezha Probe, triple-network IPs, proxy and network connectivity

This section will be relatively miscellaneous, but all centered around personal server and network asset management.

macOS

The macOS category mainly records my own Mac usage, troubleshooting, and hardware inspection.

For example:

• Hardware inspection after receiving the Mac mini M4
• VS Code auto-update failure on external APFS volume
• External drives, app migration, system log investigation

These articles lean toward personal experience, but many pitfalls are common. Especially on macOS, issues with cross-volume operations, permissions, temporary directories, and app auto-updates are usually not obvious — until something goes wrong, and then they're hard to pinpoint.

Windows and Security

The Windows category currently covers WSL, Kali, and Windows environment-related content.

Planned additions:

• WSL migration and tinkering
• Kali tool environment
• Windows Server operations experience
• Security testing environment setup

This differs from the Windows troubleshooting in the AWS category: the AWS category focuses more on cloud cases, while the Windows category is more about local environments and personal use.

Photography

Photography is another long-term section.

This section won't focus much on gear specs — it's more about the photos themselves and shooting scenes, like the Southern Anhui Sichuan-Tibet Highway, birds, and football matches. Tech blogs tend to get drier over time; photography keeps this site from being just a problem list and preserves a bit of life.

Writing Principles

I'll try to follow several principles for future articles:

• Don't include sensitive information — accounts, instance IDs, internal IPs, domains, and keys should all be generalized.
• Write less vague conclusions, more reasoning behind decisions.
• Commands should be directly copyable, but risk warnings must be included.
• Explain root causes, not just steps.
• Production operations must include backup and rollback reminders.
• Unfinished drafts shouldn't be pinned; placeholder content shouldn't be treated as published articles.

Future Plans

Short-term: clean up existing notes, especially AWS, DevOps, and VPS-related content. Medium-term: further organize the blog's visual design and navigation to make categories clearer and article lists more browsable.

This blog isn't very extensive yet, but it's already developing its own direction: technical problems don't just record "how to do it" but more importantly "why it was done this way." With long-term accumulation, it will become increasingly valuable.

1. The DeepSeek official website has been extremely popular recently and often fails to respond
   

2. Local deployment offers higher security

3. Local deployment can bypass some official restrictions

Test Environment

OS: Windows 10 Pro 22H2
CPU: AMD Ryzen 5 5600H (6C12T, Base 3.3GHz / Boost 4.2GHz)
GPU: NVIDIA GeForce RTX 3050 Ti Laptop GPU (4GB GDDR6 VRAM)
RAM: SAMSUNG 16GB DDR4-3200
IDE: LM Studio v0.3.9

Deployment Method

0. It is recommended to use an international network environment and enable TUN mode in your proxy tool

1. Download and install the LM Studio client
   Click to go

2. Change the model download directory (recommended)
   This prevents models from consuming too much space on the C drive
   

3. Download the appropriate model based on your computer configuration
   Models with Distill in the name are distilled models
   The recommended model is DeepSeek-R1-Distill-Llama-8B-Abliterated-GGUF, which runs smoothly with 4GB of VRAM. This model also removes some of DeepSeek's built-in restrictions, allowing more freedom in local use
   

4. Load the model
   After the download completes, click the top bar to load the model you just downloaded
   It is recommended to increase GPU Offload and enable Fast Attention for better performance
   

5. Start using it
   

Advanced Usage

Download and Install Cherry Studio

This section will be supplemented with specific download and installation steps later.

Register a SiliconFlow Account

Click to register

Get an API Key

This project trains a small TinyStories-style GPT model from random initialization on an Apple M4 Mac mini 16GB using MLX. It is not about calling APIs or fine-tuning an existing model, but rather walking through the entire pipeline of data preparation, tokenizer, model architecture, training loop, checkpoint, and inference generation.

This write-up leans more toward an engineering retrospective: the focus is not on training a chat-capable model, but on verifying whether a personal machine can complete an end-to-end small-scale LLM training run.

Project repository: sergioperezcheco/llm-from-scratch

Project Results

The final trained model is a GPT with 44M parameters:

| Item | Result |
|

On a Windows Server 2008 EC2 instance, the CloudWatch Agent service status was normal, but monitoring metrics could never be reported to CloudWatch. The logs repeatedly showed x509: certificate signed by unknown authority, and the root cause was ultimately traced to outdated system root certificates and insufficient TLS support.

Symptoms

The CloudWatch Agent logs continuously showed errors like:

WriteToCloudWatch failure, err: RequestError: send request failed
caused by: Post https://monitoring.<region>.amazonaws.com.cn/:
x509: certificate signed by unknown authority

Also note two easily misdiagnosed points:

• ping monitoring.<region>.amazonaws.com.cn failing does not necessarily mean the service is unreachable — Interface Endpoints typically do not respond to ICMP.
• Opening the CloudWatch API endpoint in a browser and getting 404 Not Found is also normal — it is not a regular web service.

Root Cause

The root certificate store on Windows Server 2008 is too old and may lack the root certificates needed to verify AWS server certificates, such as Amazon Root CA 1. Purely internal instances that cannot access the public internet also cannot automatically pull new trusted root certificates.

Additionally, older versions of Windows Server 2008 may also lack patches that support modern TLS chains. The end result is that the CloudWatch Agent cannot complete certificate chain verification when establishing an HTTPS connection.

Resolution Steps

1. Install SHA-2 / TLS Related Patches

First, install the required security patches for Windows Server 2008, such as KB4474419. The system must be restarted after patch installation.

wusa.exe C:\Patches\windows6.1-kb4474419-v3-x64.msu /quiet /norestart
shutdown /r /t 0

2. Import Amazon Root CA 1

Download the Amazon Root CA 1 certificate:

https://www.amazontrust.com/repository/AmazonRootCA1.cer

In an internal network environment, you can first download it from a machine with public internet access, then securely copy it to the instance.

Import it into the trusted root certificate store:

certutil -addstore -f Root C:\Patches\AmazonRootCA1.cer

You can also import it via the certmgr.msc graphical interface into "Trusted Root Certification Authorities."

3. Restart the CloudWatch Agent

net stop "Amazon CloudWatch Agent"
net start "Amazon CloudWatch Agent"

Verification

Check the Agent logs to confirm that x509: certificate signed by unknown authority no longer appears.

You can also test TCP 443 connectivity:

(New-Object System.Net.Sockets.TcpClient).Connect("monitoring.<region>.amazonaws.com.cn", 443)

If the command completes without errors, TCP layer connectivity is confirmed. Ultimately, verify that metrics are being reported normally in the CloudWatch console.

Summary

When running CloudWatch Agent on older Windows Server 2008 systems in a purely internal network environment, the common issue is not VPC Endpoint configuration, but outdated system root certificates and TLS capabilities. The recommended resolution order is:

1. Confirm CloudWatch endpoint TCP 443 is reachable.
2. Install necessary system patches.
3. Manually import Amazon Root CA 1.
4. Restart the CloudWatch Agent and observe the logs.

Such legacy systems should be included in a migration plan, as they will continually encounter certificate, TLS, patch, and software compatibility issues over time.

Symptoms

Executing AWS-RunPatchBaseline scan fails, returning something like:

The find operation did not complete successfully

The HResult may be:

-2145107934

Under the same network environment, Windows Server 2016 works normally while Windows Server 2019 fails.

Troubleshooting Process

1. Verify SSM Endpoints

Test-NetConnection ssm.<region>.amazonaws.com -Port 443
Test-NetConnection ssmmessages.<region>.amazonaws.com -Port 443
Test-NetConnection ec2messages.<region>.amazonaws.com -Port 443

If using China regions or VPC Endpoints, replace with the corresponding domain names.

2. Verify Windows Update Network

Test-NetConnection sls.update.microsoft.com -Port 443
Test-NetConnection download.windowsupdate.com -Port 80

Also check if WSUS is configured:

Get-ItemProperty "HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate" -ErrorAction SilentlyContinue
Get-ItemProperty "HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU" -ErrorAction SilentlyContinue

3. Export Windows Update Log

Get-WindowsUpdateLog

If the log contains:

The server returned HTTP status code '503'
The service is temporarily overloaded
*FAILED* [80244022] Web service call

This means the Windows Update client has connected to the Microsoft service, but the server returned an unavailable status.

Root Cause

Windows Server 2019 may be accessing a specific Microsoft update service domain, such as:

fe3.delivery.mp.microsoft.com

If that service endpoint is temporarily overloaded, it returns HTTP 503, corresponding to error code:

0x80244022

At the same time, Windows Server 2016 may work fine because it accesses a different set of update service domains that are not affected.

Workarounds

1. Wait for Microsoft Service Recovery

If you've confirmed that the network, SSM, and WSUS configuration are all normal, and the error is clearly 503, the most straightforward approach is to wait for the service to recover and retry.

2. Manually Download Patches

Download the .msu from Microsoft Update Catalog, then install via script or SSM Run Command:

https://www.catalog.update.microsoft.com/

3. Pre-cache Patches

Patch Manager itself does not provide a standard "download only, don't install" mode. You can use a custom SSM document to pre-download patches and execute installation during the maintenance window.

4. Deploy WSUS

For environments with strict patch window requirements, you can deploy an on-premises WSUS server to pre-sync patches locally, reducing dependency on the public Microsoft update service.

Summary

SSM Patch Manager scan failures are not necessarily SSM issues. When troubleshooting, work through the layers:

1. Whether the SSM Agent is online.
2. Whether AWS endpoints are reachable.
3. Whether Windows Update endpoints are reachable.
4. Whether WindowsUpdate.log shows a Microsoft server-side 503.

If the logs clearly show 0x80244022 and HTTP 503, it should generally be treated as a temporary Microsoft update service unavailability — consider retrying, manual patching, pre-caching, or WSUS.

Symptoms

The patch installation phase appears successful, but upon reboot:

We couldn't complete the updates
Undoing changes
Don't turn off your computer

After entering the system, the OS Build has not increased, and reinstalling still results in repeated rollbacks.

WindowsUpdate.log may show:

Post-reboot status ... 0x800f0922

CBS.log contains:

CBS_E_INSTALLERS_FAILED
Per-User Registry Installer ... 0x80070002

CSI logs may also show historical user NTUSER.DAT unload failures.

Troubleshooting Approach

First, rule out common causes:

sfc /scannow
DISM /Online /Cleanup-Image /RestoreHealth

Also check:

• Whether SSU is already installed.
• Whether the C drive has sufficient space.
• Whether failure persists after resetting SoftwareDistribution / catroot2.

If all of these are normal, examine the CBS/CSI logs to see if failures are concentrated in the Per-User Registry Installer phase.

Root Cause

In this case, the issue was concentrated in historical user profiles. The server had multiple legacy user directories, Unknown Profiles, and even abnormally large user profiles. During the reboot phase, updates need to load or unload user registry hives, and some NTUSER.DAT files cannot be properly unloaded, causing the patch transaction to fail and triggering a rollback.

Solution

1. Back Up First

Before performing operations in production, create an AMI or snapshot. User profile cleanup carries data risk and should not be done by blindly deleting in production.

2. Clean Up Unknown Profiles

Via the GUI:

1. Run sysdm.cpl.
2. Go to the "Advanced" tab.
3. Click "Settings" in the "User Profiles" section.
4. Delete profiles with Unknown status or those confirmed to be no longer in use.

3. Clean Up Registry ProfileList If Necessary

Carefully open the registry:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList

Cross-reference the C:\Users directory with SIDs in ProfileList, and clean up invalid entries. Export a registry backup before proceeding.

4. Reinstall the Patch

After cleanup, reboot, then reinstall the target cumulative update.

5. Temporary Workaround

If profiles cannot be cleaned up immediately and a newer monthly patch has been released, you can test installing the updated cumulative patch directly. Windows cumulative updates typically include the previous month's security content, but this should only be used as a temporary workaround — the underlying issue should still be addressed through cleanup.

Summary

When Windows patches roll back during the reboot phase, don't focus solely on Windows Update. If CBS/CSI logs point to the Per-User Registry Installer and user hive unload failures, focus on examining historical user profiles.

Profile accumulation is common on servers that have been accessed by many users over a long period. It is recommended to regularly clean up obsolete profiles to avoid issues surfacing during patch windows.

Symptoms

Users receive a permission denied message when accessing the SMB share path. The administrator has already added permissions to the target domain group in the "Security" tab, but access still fails.

Key Concepts

SMB share access is controlled by two permission layers simultaneously:

• Share Permissions
• Security Permissions / NTFS Permissions

The effective permission is the intersection of both. If only NTFS permissions are configured without share permissions, users may still be denied.

Troubleshooting Steps

1. Check Share Permissions

Log in to a domain-joined Windows machine with a domain administrator account and open:

compmgmt.msc

Connect to the FSx SVM DNS name, then navigate to:

System Tools -> Shared Folders -> Shares

Find the target share, open its properties, and check the "Share Permissions" tab. Confirm that the target user or group has at least Read permission.

2. Check NTFS Permissions

Then check the "Security" tab to confirm the file system permissions also include the target user or group.

3. Refresh Kerberos Tickets

If a user was just added to a domain group, the Kerberos TGT in their current login session may still contain old group membership information.

The most reliable approach is to have the user fully log off and log back in, rather than just locking the screen or disconnecting RDP.

You can also try purging tickets:

klist purge

But in production troubleshooting, a full logoff and logon is more straightforward and reliable.

Why This Happens

After a Windows user logs in, they receive a Kerberos ticket containing group membership information. If an administrator modifies group memberships while the user is logged in, the user's existing ticket will not automatically reflect the new permissions. FSx still sees the old identity information, causing the permission check to fail.

Summary

When FSx ONTAP SMB share access is denied, investigate in this order:

1. Whether share permissions allow access.
2. Whether NTFS permissions allow access.
3. Whether the user has logged off and back in to refresh Kerberos tickets.

Only checking the "Security" tab is not enough — this is the most common mistake in SMB permission troubleshooting.

Symptoms

FSx creation fails, with the following appearing in the console or error message:

setupFileServerRole failed
Get-ADComputer : Unable to contact the server.
This may be because this server does not exist, it is currently down,
or it does not have the Active Directory Web Services running.

This error typically occurs when FSx is setting up the file server role, joining, or querying AD objects.

Root Cause

Get-ADComputer depends on Active Directory Web Services, i.e., ADWS. ADWS uses:

TCP 9389

For FSx for Windows Single-AZ 2 and Multi-AZ types, FSx needs to access the domain controller's TCP 9389. If this port is blocked by security groups, NACLs, enterprise firewalls, or cross-region network policies, file system creation will fail.

Verification Method

On a domain-joined EC2 Windows instance in the same subnet and same security group as FSx, run the AD validation tool.

Install-WindowsFeature RSAT-AD-PowerShell

Invoke-WebRequest `
  "https://docs.aws.amazon.com/fsx/latest/WindowsGuide/samples/AmazonFSxADValidation.zip" `
  -OutFile "AmazonFSxADValidation.zip"

Expand-Archive -Path "AmazonFSxADValidation.zip"
Import-Module .\AmazonFSxADValidation

$Credential = Get-Credential
$Args = @{
  DomainDNSRoot  = "example.com"
  DnsIpAddresses = @("DC_IP_1", "DC_IP_2")
  SubnetIds      = @("subnet-xxxxxxxx")
  Credential     = $Credential
}

$Result = Test-FSxADConfiguration @Args
$Result.Failures

If the output contains a TCP 9389 failure entry, you can confirm it is an ADWS port connectivity issue.

Why Single-AZ 1 Might Succeed

Single-AZ 1 has different requirements for TCP 9389 and may succeed in the same environment. This can help determine whether service account permissions, DNS, and basic AD ports are functioning properly.

If Single-AZ 1 succeeds but Single-AZ 2 / Multi-AZ fails, the troubleshooting focus should shift to TCP 9389.

Solution

Allow TCP 9389 from the FSx subnet to all domain controllers:

• FSx security group outbound rules.
• Domain controller security group inbound rules.
• Network ACL rules in both directions.
• Local or cross-region firewall policies.
• Intermediate firewalls in enterprise networks.

After allowing access, re-run the validation tool to confirm there are no failures, then recreate FSx.

Summary

When FSx for Windows fails to join a self-managed AD, don't only check common ports like 389, 445, and 88. For Single-AZ 2 and Multi-AZ, TCP 9389 is equally critical.

When you see Get-ADComputer or ADWS-related errors, prioritize verifying 9389 connectivity from the FSx subnet to all DCs.

Prerequisites

Before enabling SQL Server HA license savings, confirm the environment meets the requirements:

• Windows Server 2019 or later.
• SQL Server 2017 or later.
• An HA cluster supports only two EC2 nodes.
• Instances need to run SSM Agent.
• Instance IAM Role needs EC2 SQL HA and SSM-related permissions.

If the environment is still on Windows Server 2016, or the cluster has more than two nodes, the prerequisites for this feature are not met.

Standby Node Restrictions

To qualify for license cost reduction, the standby node must remain passive:

• It does not process incoming business traffic.
• It does not run active SQL Server workloads.
• It cannot serve as a readable secondary replica to handle read queries.
• It should not run standalone databases outside the availability group.

The core determination is simple: as long as the node is providing data services, it is no longer a pure standby.

Readable Secondary Affects Cost Reduction

When Readable Secondary is enabled in an Always On availability group, the secondary replica can be accessed by applications, reports, or manual queries. Under license logic, this constitutes active use and requires full SQL Server licensing.

Therefore, if the goal is to obtain the standby node license cost reduction, do not enable readable secondary replicas.

Does Backup Require a Readable Secondary?

No. SQL Server supports executing certain backup scenarios on non-readable secondary replicas. In other words, for full backups and log backups, there is no need to set the secondary replica as readable.

Before actual configuration, verify against the SQL Server version and availability group backup preferences.

Enablement Steps Overview

1. Confirm SSM Agent Is Online

aws ssm describe-instance-information

The instance should show PingStatus: Online.

2. Configure IAM Permissions

Attach the following to the instance profile:

• AmazonSSMManagedInstanceCore
• AWSEC2SqlHaInstancePolicy

3. Prepare SQL Credentials

By default, NT AUTHORITY\SYSTEM can be used to read SQL Server HA metadata. If the environment restricts this account, place the SQL Server credentials in Secrets Manager and specify them during enablement.

4. Enable in the EC2 Console

In the EC2 console, select the HA cluster-related instances and navigate to:

Actions -> Instance settings -> Modify SQL High Availability settings

Check the prerequisites, and enable license savings after they pass.

After enabling, you should see:

• Primary node: Active / Full license included
• Standby node: Standby / Waived

Summary

SQL Server HA standby node license cost reduction is not a simple toggle. What truly matters is that the standby node must remain passive.

If you enable a readable secondary for queries, reports, or application reads, you lose the cost reduction eligibility. For backup scenarios, prioritize using SQL Server's supported secondary replica backup capabilities rather than turning the standby node into a readable workload node.

Symptoms

After unchecking .NET Framework 4 in Server Manager's "Remove Roles and Features" and rebooting the instance:

• Server Manager won't open.
• PowerShell reports the ServerManager command is not recognized.
• Install-WindowsFeature reports the feature name doesn't exist.
• Some PowerShell management capabilities stop working.

Example error:

ServerManager : The term 'ServerManager' is not recognized

Root Cause

Windows Server's Server Manager and related PowerShell modules depend on NetFx4-OC-Package. Unchecking .NET Framework 4 Features in the GUI actually disables a batch of OC packages that depend on NetFx4.

This is not the same as the .NET Framework 4.8 runtime version. The registry may still show .NET 4.8 is present, but the Windows optional component NetFx4 has been disabled.

Additionally, NetFx3 cannot substitute for NetFx4. Running:

DISM /Online /Enable-Feature /FeatureName:NetFx3 /All

will not restore Server Manager, which depends on .NET 4.

Recovery Steps

Before proceeding, it is recommended to create an AMI backup. All commands should be run as administrator.

1. Enable NetFx4

DISM /Online /Enable-Feature /FeatureName:NetFx4 /All

2. Enable Server Manager GUI Management Components

DISM /Online /Enable-Feature /FeatureName:Server-Gui-Mgmt /All

3. Reboot the Instance

shutdown /r /t 0

Verify after rebooting.

4. Restore Other Dependent Components as Needed

If your workload uses IIS, WCF, PowerShell ISE, or DSC, enable them as needed:

DISM /Online /Enable-Feature /FeatureName:NetFx4Extended-ASPNET45 /All
DISM /Online /Enable-Feature /FeatureName:WCF-HTTP-Activation45 /All
DISM /Online /Enable-Feature /FeatureName:WCF-TCP-PortSharing45 /All
DISM /Online /Enable-Feature /FeatureName:IIS-ASPNET45 /All
DISM /Online /Enable-Feature /FeatureName:IIS-NetFxExtensibility45 /All
DISM /Online /Enable-Feature /FeatureName:MicrosoftWindowsPowerShellISE /All
DISM /Online /Enable-Feature /FeatureName:DSC-Service /All

Verification

ServerManager
Get-Command Install-WindowsFeature
DISM /Online /Get-Features | findstr /I "NetFx4 Server-Gui PowerShell"

Summary

To meet .NET version compliance requirements, the correct approach is to install .NET cumulative updates, not to disable .NET Framework 4 Features. The latter does not make .NET "safely disappear" from the system — instead, it breaks the Windows Server management toolchain.

During recovery, enabling NetFx4 alone is not enough — you must also enable Server-Gui-Mgmt and reboot.

Risk Points

If you enable BitLocker directly on the C drive and reboot, you may encounter:

• The instance status shows running, but RDP cannot connect.
• The boot phase is waiting for a BitLocker password or recovery key.
• The console shows a black screen or cannot accept input.
• The workload is unavailable for an extended period, and recovery is only possible through snapshot/AMI rollback.

Therefore, you must first verify that the EC2 serial console is available.

Pre-Operation Preparation

• Create an AMI or EBS snapshot.
• Perform a complete rehearsal on a test instance.
• Record and securely store the recovery key offline.
• Confirm acceptance of a reboot and brief downtime window.

Enable Serial Console SAC

Run the following in an administrator PowerShell session:

bcdedit /ems '{current}' on
bcdedit /emssettings EMSPORT:1 EMSBAUDRATE:115200

bcdedit /set '{bootmgr}' displaybootmenu yes
bcdedit /set '{bootmgr}' timeout 15
bcdedit /set '{bootmgr}' bootems yes

shutdown -r -t 0

After rebooting, confirm through the EC2 console via "Connect -> EC2 Serial Console" that you can access the boot interface. If the serial console is not available, do not proceed with encrypting the system drive.

Install BitLocker Feature

Add BitLocker via Server Manager, or use PowerShell:

Install-WindowsFeature BitLocker -IncludeAllSubFeature -IncludeManagementTools
Restart-Computer

Configure Boot Without TPM

Run gpedit.msc and navigate to:

Computer Configuration -> Administrative Templates -> Windows Components -> BitLocker Drive Encryption -> Operating System Drives

Enable "Require additional authentication at startup" and check "Allow BitLocker without a compatible TPM".

Encrypt Data Drives

For data drives, test first:

1. Right-click the data drive and enable BitLocker.
2. Choose password unlock.
3. Save the recovery key.
4. Configure auto-unlock after encryption completes.

Without auto-unlock, the workload may become unavailable after every reboot because the data drive is locked.

Encrypt the System Drive

Enable BitLocker on the C drive, choose password unlock, save the recovery key, and run the BitLocker system check.

After rebooting:

1. RDP is temporarily unavailable.
2. Connect to the EC2 serial console.
3. At the black screen, enter the BitLocker password and press Enter.
4. Windows unlocks and continues booting.
5. Restore RDP after the system finishes booting.

Summary

The key to using BitLocker on EC2 Windows is not clicking "Enable Encryption" — it's whether you can unlock during the boot phase.

You must:

• Create an AMI/snapshot before encrypting.
• Enable and verify the EC2 serial console first.
• Save the recovery key.
• Configure auto-unlock for data drives.
• Rehearse the complete reboot process in a test environment first.

If you only need static EBS encryption in the cloud, prioritize using EBS encryption. BitLocker is more suitable for BYOL or specific compliance scenarios.

Symptoms

• Instance is in running state.
• System status checks and instance status checks pass.
• Security group allows port 3389.
• Instance responds to ping.
• RDP cannot connect.
• Stop & Start resolves the issue.

This behavior closely resembles a network issue, but the logs may point to insufficient system resources.

Key Logs

The event log may contain:

Not enough storage is available to process this command.

And:

System.OutOfMemoryException

You may also see Winlogon-related events, such as Winlogon crashing or failing to create a login session.

Here, "storage" does not necessarily refer to disk — in the context of Windows error codes, it can also mean insufficient memory or system resources.

Root Cause

When instance memory and page file resources are exhausted, the system cannot allocate resources for critical processes. Components required for RDP login — such as Winlogon, LSASS, and Remote Desktop Services — may fail to function properly.

Stop & Start clears the memory state, so the issue temporarily disappears, but if the instance size or application memory usage is not addressed, the problem will recur.

Troubleshooting Direction

1. Check Event Logs

Focus on the period around the failure in:

• Application.evtx
• System.evtx
• Setup.evtx

Look for:

• OutOfMemoryException
• Not enough storage is available
• Winlogon errors
• Security software or monitoring agent anomalies
• Windows Update-related anomalies

2. Check Instance Size

Confirm whether the current instance memory meets peak workload demands. If it is consistently near the upper limit, upgrade the instance size or optimize the application.

3. Deploy OS Metrics Monitoring

CloudWatch does not collect Windows memory metrics by default. You need to install the CloudWatch Agent to collect:

• Memory utilization
• Pagefile utilization
• Disk utilization
• Key process metrics

And set up alarms, such as alerting when memory utilization exceeds 85%.

Recommended Actions

Short-Term Recovery

Stop & Start can release memory and temporarily restore login capability. But this is not a permanent fix.

Medium-Term Optimization

Investigate applications, monitoring agents, and security software that consume high memory to determine if there are memory leaks or overly heavy configurations.

Long-Term Solution

If peak workloads genuinely require more memory, upgrade to a larger instance size. Before upgrading, create an AMI and confirm instance type compatibility with ENA/NVMe.

Summary

RDP login failure is not necessarily a port 3389, security group, or NACL issue. As long as the instance is still pingable and status checks pass, you should also examine the Windows event logs.

If the logs show OOM, insufficient system resources, and Winlogon anomalies, the root cause is most likely memory exhaustion. Stop & Start only provides temporary recovery — long-term, you need to monitor memory and adjust instance size or application configuration.

Symptoms

Client access attempt:

\\<server-private-ip>\share

Cannot open, and no authentication prompt appears. Command line execution:

net use Z: \\<server-private-ip>\share

Reports:

System error 1792 has occurred.
The attempt to logon to the network account failed because the network logon service is not started.

Afterward, port 445 connection timeouts and ping failures may even occur.

Troubleshooting Process

1. Rule Out AWS Network Issues First

Confirm:

• Both instances are in the same VPC or have routable connectivity.
• Security group allows TCP 445.
• NACL is not blocking.
• Route table has a local or correct route.
• Windows Defender Firewall policy is not blocking.

2. Packet Capture on Both Ends

Packet capture shows the SMB protocol has already begun negotiation:

SMB2 Negotiate Protocol Request
SMB2 Negotiate Protocol Response
SMB2 Session Setup Request
SMB2 Session Setup Response: STATUS_NETLOGON_NOT_STARTED

This indicates traffic has reached the server — it is not an AWS underlying network packet drop.

3. Understanding Netlogon

In a Workgroup environment, the Netlogon service not running by default is normal. Local account SMB authentication typically completes through local SAM + NTLM, and should not necessarily fail just because Netlogon is stopped.

If STATUS_NETLOGON_NOT_STARTED is returned and then all traffic is blocked, suspect that security software or EDR is intercepting SMB authentication traffic.

Solution

1. Temporarily Disable EDR to Verify

Temporarily disable security software during a change window to verify whether SMB access is restored. If port 445 and ping both recover after disabling, the root cause is essentially confirmed.

2. Use Explicit Local Account Authentication

In a Workgroup environment, do not rely on implicit credentials. Use a server local account:

net use Z: \\<server-private-ip>\share /user:<server-computer-name>\Administrator

Note that <server-computer-name> must be the server's computer name, not the client's.

3. Adjust EDR Policy

Contact the security software vendor or security team to whitelist normal SMB/NTLM authentication traffic and prevent false positives.

4. Long-Term Recommendation: Join a Domain

If multiple Windows instances frequently share files, consider joining Active Directory and using domain accounts with group-based permissions to reduce the complexity of Workgroup + local account authentication.

Summary

When EC2 Windows SMB access fails, don't only look at security groups. If packet capture already shows SMB protocol negotiation and authentication-stage errors, the issue has moved into the OS or security software layer.

STATUS_NETLOGON_NOT_STARTED in a Workgroup scenario does not necessarily mean Netlogon itself is the root cause. Combined with subsequent traffic being blocked, EDR or security software interception should be a primary investigation focus.

Why VM Export Fails

Running create-instance-export-task may produce:

An error occurred (NotExportable) when calling the CreateInstanceExportTask operation:
The image ID (ami-xxxxxxxx) provided contains AWS-licensed software and is not exportable.

This is a product limitation, not an IAM permissions issue. Windows / SQL Server / Marketplace images launched from public AWS AMIs are generally non-exportable.

Alternatives

Available options:

• If you can log in to the instance: use Disk2vhd.
• If you don't want to spin up a rescue instance: use coldsnap + qemu-img.
• If the instance can be stopped or has crashed: use a rescue instance for block-level EBS reading, then qemu-img conversion.

This article documents the third approach.

Key Pitfall: Storage Controller Drivers

EC2 Windows typically runs on NVMe devices. After exporting to VMware, the controller may change to LSI Logic SAS, SATA, or IDE. If these drivers are not set to Boot start in the Windows registry, you'll get a blue screen at boot:

UNMOUNTABLE_BOOT_VOLUME

So before conversion, you need to offline-modify the SYSTEM hive and set the Start value of common storage drivers to 0.

Procedure

1. Stop Instance and Detach Volume

aws ec2 stop-instances --instance-ids <instance-id>
aws ec2 wait instance-stopped --instance-ids <instance-id>

aws ec2 detach-volume --volume-id <volume-id>
aws ec2 wait volume-available --volume-ids <volume-id>

aws ec2 attach-volume \
  --volume-id <volume-id> \
  --instance-id <helper-instance-id> \
  --device /dev/sdg

It's recommended to create a snapshot of the volume before proceeding.

2. Identify Disk on Rescue Instance

sudo lsblk -o NAME,SIZE,SERIAL,MOUNTPOINT,FSTYPE

On Nitro instances, EBS appears as /dev/nvmeXn1. You can match volume IDs via the SERIAL field.

3. Mount Working Disk

sudo mkfs.xfs -f /dev/nvme1n1
sudo mkdir -p /mnt/work
sudo mount /dev/nvme1n1 /mnt/work

The working disk must be larger than the actual output size of the target volume.

4. Install Tools

sudo dnf install -y qemu-img gcc make perl

If hivex is not available in the repository, you'll need to install it from source. It's used to edit Windows registry hives.

5. Mount NTFS and Backup SYSTEM Hive

sudo modprobe ntfs3
sudo mkdir -p /mnt/windows
sudo mount -t ntfs3 -o rw /dev/nvme2n1p1 /mnt/windows

sudo cp /mnt/windows/Windows/System32/config/SYSTEM /mnt/work/SYSTEM_BACKUP
sudo cp /mnt/windows/Windows/System32/config/SYSTEM /tmp/SYSTEM_EDIT

6. Enable Generic Storage Drivers

The following services need their Start set to 0:

| Service | Purpose |
|

Project repository: bin456789/reinstall

Reference documentation: CSDN original article

Steps

Why Graviton Can't Officially Run Windows

AWS only provides Windows AMIs for the x86 architecture. Graviton uses the ARM64 architecture. Although Microsoft has Windows on ARM, AWS has not done official driver adaptation and AMI publishing for the Graviton platform. So the normal flow is: choose t3/t3a x86 instances for Windows, and Graviton can only run Linux.

But "not officially supported" doesn't mean "can't run." The reinstall.sh script can directly DD-install Windows on an existing Linux instance, including the ARM64 version. Test environment:

• Instance type: t4g.large (2 vCPU / 8 GB)
• Original system: Amazon Linux 2023 aarch64, 50 GB EBS (gp3), UEFI boot
• Target system: Windows 11 Pro 25H2 ARM64 (Build 26200.6584), Chinese version

Core Finding: The Counter-Intuitive NVMe Driver Pitfall

The most counter-intuitive part of the entire process: Injecting the official AWS NVMe driver actually causes boot failure; only the Microsoft inbox StorNVMe works properly.

StorNVMe.sys is a standard NVMe driver built into the system image by Microsoft since Windows 8.1. It loads automatically during installation without any external injection or manual operation. The Has StorNVMe: true in the reinstall.sh log means the script detected the ISO includes this inbox driver.

Real-world comparison:

| Approach | NVMe Driver | ENA Driver | Result |
|

Symptoms

Accessing an internal service results in an error:

curl https://service.example.internal/api/v1/health

curl: (60) SSL certificate problem: unable to get local issuer certificate

The architecture is roughly:

Client -> Internal ALB 443 -> Backend Target 443

If bypassing the ALB and directly accessing the backend target succeeds, it indicates the backend service itself is most likely not the root cause.

Troubleshooting Approach

1. Don't Rely on Public SSL Check Results

If the domain resolves to an internal ALB in a Route 53 private hosted zone, a public SSL checker may see a different public DNS record. Public check results cannot represent the certificate actually served by the internal ALB.

2. Use openssl to Inspect the Actual ALB Certificate

Run the following on an internal client:

openssl s_client -showcerts \
  -connect <alb-dns-name>:443 \
  -servername service.example.internal </dev/null

Focus on two things:

• Whether the certificate SAN/CN covers the accessed domain.
• Whether the intermediate certificate served by the ALB matches the site certificate's signing chain.

3. Distinguish Between Two Problems

A broken certificate chain and a domain mismatch are two different problems:

• A broken chain causes the client to fail to find a trusted parent CA.
• SAN not covering the domain causes hostname verification to fail.

Either one can cause HTTPS to fail.

Root Cause

In this case, the site certificate was issued by a DV intermediate CA, but the one included during ACM import was a different OV intermediate CA. The ALB therefore served a mismatched intermediate certificate chain to the client, and curl could not build a complete trust chain.

At the same time, the ALB-bound certificate's SAN also did not cover the actual domain being accessed. In other words, even if the certificate chain were fixed, it would still fail due to the domain mismatch.

Solution

1. Fix the ACM Certificate Chain

Re-import the original ACM certificate and upload the correct intermediate certificate chain:

aws acm import-certificate \
  --certificate-arn <certificate-arn> \
  --certificate fileb://site.crt \
  --private-key fileb://site.key \
  --certificate-chain fileb://correct-chain.crt

The advantage of using reimport is that it preserves the original ARN, so the ALB listener does not need to re-select a certificate.

2. Ensure the Certificate Covers the Accessed Domain

Check whether the certificate SAN includes the domain being used:

openssl x509 -in site.crt -noout -text | grep -A1 "Subject Alternative Name"

If it does not, you need to request or import a new certificate that covers the target domain and bind it to the ALB HTTPS listener.

3. Clean Up ALB Listener Certificates

If the ALB has multiple certificates attached, it is recommended to clean up unused certificates to avoid SNI matching confusion and operational judgment errors.

Summary

When ALB HTTPS reports unable to get local issuer certificate, the troubleshooting focus should be the certificate chain actually served by the ALB, not the backend Nginx or public SSL check results.

Recommended fixed troubleshooting order:

1. Use openssl s_client -showcerts on an internal client to inspect the certificate served by the ALB.
2. Check whether SAN/CN covers the accessed domain.
3. Check whether the intermediate certificate matches the site certificate's signing chain.
4. Use ACM reimport to fix the certificate chain, and replace the listener certificate if necessary.

Scenario

The goal is to count the following in a versioning-enabled S3 bucket:

• Number of objects with a specified file extension.
• Total size of objects with a specified file extension.
• Including all historical versions, not just the current version.

If the business requires "results right now," S3 Inventory may not be suitable because it is an asynchronous report and the first generation typically has a delay.

Why Use list-object-versions

The regular list-objects only looks at the current object version and cannot cover historical versions. For buckets with versioning enabled, use:

aws s3api list-object-versions

Then use --query 'Versions[*].[Key, Size]' to extract only the object Key and Size, reducing downstream processing cost.

Counting Command

The following example counts .jpg and .png objects:

aws s3api list-object-versions \
  --bucket <bucket-name> \
  --region <region> \
  --query 'Versions[*].[Key, Size]' \
  --output text |
grep -Ei "\.(jpg|png)[[:space:]]+[0-9]+$" |
awk '
BEGIN {
  fmt = "Total image objects (including historical versions): %d\n"
}
{
  count++;
  size += $NF;
}
END {
  print "======================";
  printf fmt, count;
  printf "Total size (including historical versions): %.2f GB\n", size/1024/1024/1024;
  print "======================";
}'

Example output:

======================
Total image objects (including historical versions): 120446
Total size (including historical versions): 56.33 GB
======================

Notes

• It is recommended to execute this on an EC2 instance in the same region to reduce network latency.
• If there are many objects, CLI calls will incur List request costs.
• If you only need the current version, do not use list-object-versions — use list-objects-v2 instead.
• If the object scale is very large and some delay is acceptable, S3 Inventory is more suitable for periodic reporting.
• If keys contain special characters such as newlines, text pipeline processing will have edge-case issues. For rigorous scenarios, use JSON + jq.

Summary

When you need to urgently count objects with specific extensions in an S3 bucket, list-object-versions + grep + awk is a simple and effective solution. Its advantages are that it is real-time, lightweight, and requires no waiting for Inventory; its disadvantage is that it is more suited for one-off counting, not long-term periodic reporting.

Scenario

In a customer environment, many domain users log into the AWS console via SAML AssumeRole, potentially using high-privilege roles like PowerUser or Administrator.

Due to compliance requirements, certain users need to be restricted from accessing EC2 instances through the console, primarily involving two capabilities:

• SSM Session Manager: ssm:StartSession
• EC2 Instance Connect: ec2-instance-connect:SendSSHPublicKey

Why Not Use RoleSessionName

After SAML federated login, IAM generates a session identifier. The more stable and available global condition key for policy evaluation is aws:userid. It typically contains the role ID and session name, in a format like:

<role-id>:<session-name>

Because the role ID portion changes with the Role, it is not suitable for hardcoding. Instead, you can use a wildcard to match the session suffix:

*:user-or-ou-id

Example Policy

Add an explicit Deny to the target IAM Role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyEC2LoginForSpecificFederatedUsers",
      "Effect": "Deny",
      "Action": [
        "ssm:StartSession",
        "ec2-instance-connect:SendSSHPublicKey"
      ],
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "aws:userid": [
            "*:<user-or-ou-id>"
          ]
        }
      }
    }
  ]
}

Explicit Deny takes priority over Allow, so even if the Role already has high-privilege policies, users matching the condition will still be blocked.

Verification Method

1. Log into the console as the target SAML user.
2. Attempt to connect to EC2 via Session Manager.
3. Attempt to use EC2 Instance Connect.
4. Then verify that unrestricted users are not affected.

If the Deny is working, the target user will be denied when calling the related APIs.

Recommended Long-Term Approach

Hardcoding user suffixes based on aws:userid can quickly solve the problem, but the maintenance cost is high. A better approach is to split permissions at the identity source level:

• Create a restricted user group in AD, e.g., AWS-No-EC2-Login.
• Map different user groups to different IAM Roles in the IdP.
• Create a dedicated restricted Role that does not grant EC2 login capabilities.

This makes permission boundaries clearer and easier to audit and automate.

Summary

If you need to temporarily restrict certain SAML users from logging into EC2 via the console, you can use aws:userid + explicit Deny to precisely block ssm:StartSession and ec2-instance-connect:SendSSHPublicKey.

In the long term, it is recommended to manage user grouping and Role mapping at the identity source side, rather than maintaining increasingly complex conditional policies within a single high-privilege Role.

Symptoms

Executing partition expansion:

sudo growpart /dev/nvme0n1 1

Returns an error like:

failed [sfd_list:1] sfdisk --list --unit=S /dev/nvme0n1
FAILED: failed: sfdisk --list /dev/nvme0n1

The environment is typically:

• Nitro architecture instance.
• Amazon Linux 2.
• Root filesystem is XFS.
• Device name is /dev/nvme0n1, root partition is /dev/nvme0n1p1.
• / is at or near 100%.

Root Cause

growpart needs to create temporary files in system directories and rewrite the partition table when executing. If the root filesystem is so full that only a few tens of KB remain, the underlying sfdisk call may not complete successfully, resulting in partition expansion failure.

In this case, the first priority is not to keep running growpart repeatedly, but to first free up some space on the root partition.

Resolution Steps

1. Clean the yum Cache

On Amazon Linux 2, you can start by cleaning the yum cache:

sudo yum clean all

Confirm the root partition has at least a few MB of free space:

df -h /

2. Expand the Partition

sudo growpart /dev/nvme0n1 1

3. Expand the XFS Filesystem

XFS requires using the mount point for expansion:

sudo xfs_growfs -d /

For ext4, you should use:

sudo resize2fs /dev/nvme0n1p1

4. Verify

df -hT /
lsblk

Confirm the root partition size and available space have been updated.

Notes

• After modifying the volume size in the EBS console, wait for the status to complete before proceeding with in-system expansion.
• Individual EBS volumes have modification frequency limits — do not repeatedly attempt expansion.
• When the root partition is nearly full, it is recommended to prioritize cleaning caches, old logs, and temporary files.
• For the long term, it is recommended to use CloudWatch Agent to collect disk usage metrics and set up advance alerts.

Summary

growpart reporting an sfdisk failure does not necessarily mean the partition table is corrupted — it may simply be that the root filesystem has no space for the tool to operate. For root volume expansion on Amazon Linux 2 + XFS, follow this order:

1. Free up a small amount of space.
2. Expand the partition with growpart.
3. Expand the filesystem with xfs_growfs.
4. Verify with df -hT.

Symptoms

The failures fall into two categories:

1. Instance fails to boot, and CloudTrail shows KMS CreateGrant or Decrypt permission errors.
2. Instance attempts to boot, but the system log is stuck at:

Give root password for maintenance
(or press Control-D to continue):

The second case causes the OS to fail to fully start, the instance cannot respond to underlying health checks, and the final manifestation is an instance status check failure.

Problem 1: Insufficient KMS Permissions

If the EBS volume uses a KMS CMK for encryption, the IAM role used to launch the instance must have permissions to use that KMS key. At minimum, the following are required:

{
  "Effect": "Allow",
  "Action": [
    "kms:Decrypt",
    "kms:GenerateDataKey",
    "kms:CreateGrant"
  ],
  "Resource": "arn:aws-cn:kms:<region>:<account-id>:key/<key-id>"
}

If the role lacks permissions, EC2 cannot decrypt the system or data volumes during the boot phase, and the instance will fail to start.

Problem 2: fstab Blocking Boot

On Linux, NVMe device names may change with reboots or underlying changes. If /etc/fstab contains a hardcoded entry like:

/dev/nvme2n1p1 /data ext4 defaults 1 2

When the device name changes or the volume does not exist, the system will wait for the mount during boot and eventually enter emergency / maintenance mode.

A more stable approach is to use UUID with nofail:

UUID=<volume-uuid> /data ext4 defaults,nofail 1 2

Network filesystems should also add _netdev:

server:/share /mnt/share nfs defaults,_netdev,nofail 0 0

Fixing fstab via a Rescue Instance

1. Prepare a Rescue Instance

Launch a Linux rescue instance in the same availability zone. Stop the original instance, detach the original root volume, and attach it to the rescue instance.

2. Mount the Original System Root Partition

If the original system uses LVM, first install the tools and activate the volume group:

sudo dnf install lvm2 -y
sudo vgscan
sudo vgchange -ay
sudo lvs

Mount the original root partition:

sudo mkdir -p /mnt/rescue
sudo mount -o nouuid /dev/<vg-name>/<root-lv> /mnt/rescue

3. Modify fstab

sudo vi /mnt/rescue/etc/fstab

First comment out the risky data volume mount entries so the system can boot. After booting, use lsblk -f to get the UUID and switch to a stable configuration.

4. Unmount and Reattach to the Original Instance

sudo umount /mnt/rescue
sudo vgchange -an

Then in the console, reattach the root volume to the original instance, start it, and verify.

Verification

After the instance boots, execute:

lsblk -f
sudo systemctl daemon-reload
sudo mount -a
df -h

If mount -a completes without errors, the fstab configuration is essentially correct.

Summary

EC2 Linux boot failures should be distinguished at two layers:

• AWS control plane: KMS, IAM, EBS status, volume attachment relationships.
• OS internals: fstab, LVM, filesystem, network mounts.

For encrypted volume boot failures, prioritize checking CloudTrail and KMS permissions; for systems stuck in maintenance mode, prioritize checking the console system log and /etc/fstab. For data volume mounts, it is recommended to consistently use UUID + nofail to prevent device name changes from blocking system startup.

Symptoms

When manually installing a specific Windows Server 2019 cumulative update, the installer returns:

The update is not applicable to your computer

The CBS log may contain:

Higher version found for package ..., superseded.

WindowsUpdate.log may show:

The volatile RebootRequired key exists

Analysis Approach

1. Compare OS Build

First, check the current system version:

winver

Or read from the registry:

Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion" |
  Select-Object CurrentBuildNumber, UBR

If the current Build is already higher than the version corresponding to the target KB, it means the KB content has been superseded by a newer cumulative update, and "not applicable" is the expected result.

2. Check Whether CBS Has Installed a Newer Package

The Higher version found entry in CBS.log is critical. Windows cumulative updates have a supersession relationship — a newer LCU already contains the content of older LCUs.

3. Check Whether a Reboot Is Required

If RebootRequired appears in WindowsUpdate.log, it means the system may have completed a staged installation, but the Build/UBR values in the registry haven't been updated during the reboot process.

This can cause winver to still show the old version while CBS already shows a newer package exists.

Solution

1. Do Not Repeatedly Install the Old KB

If CBS already shows a newer package exists, stop trying to install the old KB to avoid wasting maintenance windows.

2. Reboot During a Maintenance Window

In production environments, it is recommended to create an AMI backup first, then reboot the instance:

Restart-Computer

After rebooting, verify again:

winver
Get-HotFix | Sort-Object InstalledOn -Descending | Select-Object -First 10

3. Prioritize Installing the Latest Cumulative Update

If there is no compliance requirement to install a specific old KB, it is recommended to install the latest LCU directly. Windows cumulative updates typically include all previous update content.

Summary

"The update is not applicable to your computer" does not necessarily mean failure. When troubleshooting, don't rely solely on winver — also consider:

• The OS Build corresponding to the target KB.
• Whether CBS.log already shows a newer package exists.
• Whether RebootRequired exists in WindowsUpdate.log.

If the system has already installed a newer update but hasn't rebooted, the correct action is usually to schedule a maintenance window reboot, rather than repeatedly trying to manually install the old patch.

Background

Some .NET cumulative updates are container packages that contain multiple individually applicable .msu files. For example, a parent KB might embed:

• An MSU for .NET 3.5 + 4.7.2.
• An MSU for .NET 3.5 + 4.8.

If the system is already running .NET 4.8, only the one corresponding to 4.8 needs to be installed.

Key Logs

WindowsUpdate.log repeatedly shows:

ProtocolTalker  *FAILED* [C8000402] PopulateDataStore failed
ProtocolTalker  *FAILED* [C8000402] Sync of Updates
Agent           * END * Finding updates ... Exit code = 0xC8000402

Meanwhile, CBS.log has no trace of the target KB.

This indicates the patch hasn't even entered the CBS installation engine — the failure occurs at the earlier WUA scan stage.

Root Cause Determination

The flow of wusa.exe installing an .msu is roughly:

Double-click .msu -> wusa.exe -> WUA applicability scan -> CBS installation

If WUA's DataStore is corrupted, or if repeated registration of offline scan sources causes metadata pollution, WUA may fail at the PopulateDataStore stage, ultimately returning "0 updates found" or an installation failure.

At this point, continuing to double-click .msu is pointless, since execution never reaches CBS.

Solution: Extract and Inject CAB Using DISM

The core idea is to bypass WUA and let CBS handle the CAB package directly.

1. Extract the MSU

$msu = "C:\Patches\windows10.0-kbxxxxx-x64-ndp48.msu"
$dst = "C:\Patches\Extract"
New-Item -ItemType Directory -Force -Path $dst | Out-Null
expand.exe -F:* $msu $dst

2. Install the CAB Using DISM

$cab = Get-ChildItem $dst -Filter "Windows10.0-KB*.cab" | Select-Object -First 1
DISM.exe /Online /Add-Package /PackagePath:"$($cab.FullName)" /NoRestart /LogPath:C:\Patches\dism.log

3. Reboot

shutdown /r /t 30

4. Verify

Get-HotFix -Id KBxxxxx
DISM.exe /Online /Get-Packages | findstr /I "DotNetRollup"

For .NET updates, you can also check whether key .NET file versions have been updated.

Optional: Repair the WUA Scan Channel

If you need Windows Update scanning to work normally going forward, you can reset the WUA database:

Stop-Service -Name wuauserv, BITS, cryptSvc -Force
Rename-Item C:\Windows\SoftwareDistribution C:\Windows\SoftwareDistribution.old
Rename-Item C:\Windows\System32\catroot2 C:\Windows\System32\catroot2.old

$svc = New-Object -ComObject Microsoft.Update.ServiceManager
$svc.Services | Where-Object { $_.IsScanPackageService } |
  ForEach-Object { $svc.RemoveService($_.ServiceID) }

Start-Service -Name wuauserv, BITS, cryptSvc

This is not a required step for installing the patch — it only restores WUA scanning capability.

Summary

If the target KB has no trace at all in CBS.log and WindowsUpdate.log points to 0xC8000402 PopulateDataStore failed, the issue should be located in the WUA scan layer.

The approach is:

1. Confirm the applicable embedded MSU.
2. Extract the MSU to obtain the CAB.
3. Inject it directly using DISM /Add-Package.
4. Reboot and verify.

This method is suitable for scenarios where WUA is corrupted but CBS is still functioning normally.

Background

new-api was originally running in Docker, using an SQLite database by default. As logs, users, and channel data grew, the SQLite database size approached 200MB, making it unsuitable for long-term maintenance in a container data volume.

My goals were:

• Start a MySQL container on the 1Panel-managed server
• Fully import new-api's SQLite data into MySQL
• Test with the new container first, then switch reverse proxy traffic
• Minimize impact on production services

The most important principle in production migration is: don't directly stop the old service. Bring up the new environment first, confirm everything is working, then switch traffic.

iptables and Docker Port Mapping Issue

Error Symptom

When installing a MySQL container via 1Panel, the container can be created successfully, but fails during startup with an error like:

iptables: No chain/target/match by that name.

Looking further at Docker-related logs, you can see the nftables backend incompatibility:

iptables v1.8.9 (nf_tables): chain `DOCKER' in table `filter' is incompatible, use 'nft' tool.

Root Cause

Debian 12 bookworm defaults to iptables-nft, which is the nftables backend. Docker maintains its own DOCKER chain in iptables for port mapping, but in this environment, the chain state in the filter table is incompatible with the nft backend.

There's a subtle trap here: existing containers' port mappings may still work normally because the NAT table isn't immediately affected. What actually fails is new containers or newly added port mappings.

Fix

First back up the current iptables rules:

mkdir -p /root/backup-iptables-$(date +%Y%m%d)
iptables-save > /root/backup-iptables-$(date +%Y%m%d)/iptables-all.txt

Then switch iptables and ip6tables to legacy mode:

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

Finally restart Docker:

systemctl restart docker

If containers have restart: always or unless-stopped configured, they will automatically recover after Docker restarts. Containers using host network mode, such as openresty, are generally unaffected by this port mapping issue.

SQLite to MySQL Migration Challenges

new-api uses GORM, which can auto-create tables when first connecting to MySQL, but this doesn't mean it will auto-migrate SQLite data. The actual migration requires converting the SQLite dump into MySQL-importable SQL.

Direct conversion encounters several types of problems:

1. SQLite allows TEXT fields in primary keys or indexes; MySQL throws ERROR 1170.
2. SQLite supports partial indexes, e.g., WHERE deleted_at IS NULL, which MySQL can't directly replicate.
3. SQLite BLOB byte strings may be written as X'...', which MySQL JSON columns can't directly accept.
4. MySQL doesn't allow TEXT or BLOB fields to have default values.
5. SQLite is case-sensitive by default; MySQL's default collation is usually case-insensitive, which may cause unique key conflicts.

So this step can't rely solely on simple search-and-replace; a conversion script is more suitable for handling the structure and data.

Conversion Script Strategy

I ultimately used a script to generate MySQL-importable SQL, performing these transformations:

• TEXT fields involved in indexes, primary keys, or unique keys changed to VARCHAR(191)
• Non-indexed large text fields changed to LONGTEXT
• SQLite partial indexes converted to MySQL-compatible expression indexes
• BLOB JSON byte strings converted to UTF-8 JSON strings
• SQLite timestamps with timezone converted to MySQL DATETIME(6) literals
• All using utf8mb4_bin collation to preserve SQLite's case-sensitive behavior as much as possible

Don't write actual database passwords, root passwords, or DSNs in plaintext in public articles or repositories. Sensitive fields in the commands below are replaced with placeholders.

Blue-Green Deployment Flow

The overall process is: import data first, start the new container for testing, then switch the reverse proxy.

1. Import MySQL Data

First create the target database and user, and grant permissions:

CREATE DATABASE `one-api` CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
CREATE USER 'one-api'@'%' IDENTIFIED BY '<mysql-user-password>';
GRANT ALL PRIVILEGES ON `one-api`.* TO 'one-api'@'%';
FLUSH PRIVILEGES;

Then import the converted SQL:

mysql -u one-api -p one-api < /root/one-api-mysql-importable.sql

If you've already had new-api connect to an empty MySQL, GORM may have inserted default data. In that case, you need to clear the default data first to avoid primary key or unique key conflicts during import.

2. Start MySQL Version new-api Container

The new container listens on 3001 first, while the old container continues on 3000:

docker run -d --name new-api-mysql \
  --network 1panel-network \
  --restart unless-stopped \
  -p 3001:3000 \
  -e SQL_DSN="one-api:<mysql-user-password>@tcp(<mysql-host>:3306)/one-api?charset=utf8mb4&parseTime=True&loc=Local" \
  -e TZ=Asia/Shanghai \
  -e ERROR_LOG_ENABLED=true \
  -e BATCH_UPDATE_ENABLED=true \
  -v /home/ubuntu/data/new-api-mysql:/app/data \
  calciumion/new-api:latest --log-dir /app/logs

Two things to note here:

• The new-api container must join a Docker network that can access MySQL
• SQL_DSN should include charset=utf8mb4&parseTime=True&loc=Local

3. Test the New Container

First confirm the service status via API:

curl http://localhost:3001/api/status

It should return success: true under normal circumstances. Then check whether the data volume of core tables (users, channels, configurations, logs) matches the SQLite version.

The data volume after my migration was approximately:

| Table | Row Count |
|

Symptoms

After Docker container creation completes, the startup phase throws an error:

Error response from daemon: failed to set up container networking:
driver failed programming external connectivity on endpoint ...
Unable to enable OPEN PORT rule:
iptables: No chain/target/match by that name. (exit status 1)

This error can be misleading. The container itself isn't failing to create — the port mapping setup is failing during the Starting phase.

Root Cause

Debian 12 bookworm defaults to iptables-nft, which is the nftables backend. Docker maintains its own DOCKER chain in iptables for bridge networking and port mapping.

When the DOCKER chain in the filter table is incompatible with the nft backend state, you get errors like:

iptables v1.8.9 (nf_tables): chain `DOCKER' in table `filter' is incompatible, use 'nft' tool.

A key detail: the DOCKER chain in the NAT table may still be working normally, so existing port mappings won't necessarily break immediately. What's actually affected is new containers or newly added port mappings.

Quick Diagnosis

First check which backend iptables is currently using:

update-alternatives --query iptables | grep -E 'Status|Value'

Then check the Docker chain in the filter table:

iptables -L DOCKER -n

If you see incompatible or the chain doesn't exist, check the NAT table status:

iptables -t nat -L DOCKER -n

Finally confirm current containers and port mappings:

docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'

If existing containers are still running normally but new ones won't start, you can pretty much pin it on Docker port mapping and iptables backend compatibility.

Fix Steps

1. Back Up Existing Rules

Always back up before modifying network rules, so you have something to roll back to:

mkdir -p /root/backup-iptables-$(date +%Y%m%d)
iptables-save > /root/backup-iptables-$(date +%Y%m%d)/iptables-all.txt
iptables-save -t nat > /root/backup-iptables-$(date +%Y%m%d)/iptables-nat.txt
docker ps --format '{{.Names}} {{.Image}} {{.Status}}' > /root/backup-iptables-$(date +%Y%m%d)/containers.txt

2. Switch to Legacy Mode

Switch both IPv4 and IPv6 iptables to legacy:

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

3. Restart Docker

systemctl restart docker

If containers have restart: always or unless-stopped configured, Docker will automatically bring them back up.

4. Verify

Check if the DOCKER chain displays correctly:

iptables -L DOCKER -n

Confirm container status:

docker ps

Check if critical ports are listening:

ss -tlnp | grep -E ':80 |:443 |:3000 '

If it was an application that failed to install in 1Panel, try reinstalling or restarting the application at this point — it should usually recover.

Notes

• Containers using host network mode are usually unaffected by this issue, e.g., openresty listening directly on host ports.
• Containers using bridge network mode rely on iptables for port mapping — MySQL, new-api, and similar containers will be affected.
• Before fixing, confirm that important containers have a restart policy to avoid Docker restart leaving services without automatic recovery.
• If the system has firewalld enabled, you may also need to check if it has rewritten the rules.
• Don't directly flush iptables rules in production — always back up first.

Environment Record

The environment where this issue occurred was approximately:

| Item | Value |
|

When I get a new Mac, I like to do a systematic inspection first: hardware info, SSD health, port status, security configuration, and system stability — all checked through. This way, if any issues come up later, I can tell whether it's a machine problem or caused by the usage environment.

This article is based on the actual inspection process for the Mac mini M4. The commands are mainly applicable to Apple Silicon Macs.

Basic Information

View hardware overview:

system_profiler SPHardwareDataType

Focus on these fields:

• Model and model identifier
• Chip model
• Unified memory capacity
• Serial number
• System firmware version

View macOS version:

sw_vers

View system uptime:

uptime

View first setup date:

ls -la /var/db/.AppleSetupDone

The modification time of .AppleSetupDone can usually serve as a reference for when the initial setup was performed.

Storage System Inspection

Built-in SSD Information

system_profiler SPNVMeDataType

You can also check the diskutil output:

diskutil info disk0

Key things to focus on:

• SSD model
• Protocol
• TRIM support
• SMART status
• APFS container and volume status

SMART Health Data

First install smartmontools:

brew install smartmontools

View the complete SMART data for the built-in SSD:

sudo smartctl -a /dev/disk0

Common metrics can be understood as follows:

| Metric | Meaning | Normal Reference |
|

Origin

I opened my Mac mini early in the morning to start working and habitually clicked the VS Code icon in the Dock. The icon turned into a blank macOS generic app icon.

Opening Finder to the application directory also showed VS Code in an abnormal state:

My VS Code wasn't installed in the system default /Applications/, but in the applications directory on an external APFS volume.

First Reaction: Where Did the App Go?

First check the application directory:

ls -la "/Volumes/<external-volume>/applications/" | grep -i "code\|visual"

No output. In other words, the Visual Studio Code.app directory no longer exists.

But the Dock still has the old path saved:

defaults read com.apple.dock persistent-apps | grep -A2 "Visual Studio Code"

The output still shows a path like:

file:///Volumes/<external-volume>/applications/Visual%20Studio%20Code.app/

The Dock remembered an app that no longer exists, so macOS could only display the generic icon.

The Real Culprit: VS Code Auto-Update

Looking further at the system temporary directory, I found traces of VS Code's updater:

find /private/var/folders -name "*.ShipIt*" -maxdepth 4

Multiple directories like this appeared:

/private/var/folders/.../T/com.microsoft.VSCode.ShipIt.xxxxxxxx

Entering one of these directories revealed a complete Visual Studio Code.app package. In other words, the old version had been deleted from the target directory, but the new version was stuck in the temporary directory and never successfully moved back to the installation location.

Checking the version of the app in the temporary directory:

defaults read "/private/var/folders/.../ShipIt.xxxxxxxx/Visual Studio Code.app/Contents/Info.plist" CFBundleShortVersionString

Confirmed it was the updated new version.

The conclusion was clear: VS Code's auto-updater deleted the old version, but moving the new version back to the original location failed.

Why Did It Fail?

VS Code uses Electron's Squirrel.Mac for auto-updates. The core helper process is called ShipIt. The general flow is:

1. Check for updates and download new version
2. Extract to /private/var/folders/.../T/
3. Launch the ShipIt helper process
4. ShipIt waits for VS Code to exit
5. Delete the old Visual Studio Code.app
6. Move the new app to the original installation location

The problem is that the temporary directory and installation directory are on different volumes.

macOS's temporary directory is typically on the boot volume:

/private/var/folders/<hash>/T/

While my VS Code is on an external APFS volume:

/Volumes/<external-volume>/applications/Visual Studio Code.app

Same-volume moves are usually just directory entry changes, which is fast. Cross-volume moves become copy-then-delete. This process isn't atomic — as long as the copy is interrupted, the target volume has permission issues, disk space is insufficient, or the volume is briefly unavailable, you can end up in a state where "old version deleted, new version didn't arrive."

This isn't a VS Code-specific problem. Any updater that places the new version in the boot volume's temporary directory and then moves the app to another volume has similar risk.

Why Did It Disappear Again After Reinstall?

The first time, I moved the app back from the temporary directory:

mv "/private/var/folders/.../T/com.microsoft.VSCode.ShipIt.xxxxxxxx/Visual Studio Code.app" \
   "/Volumes/<external-volume>/applications/"

Then disabled auto-update in VS Code settings:

{
  "update.mode": "none"
}

I thought the problem was solved, but after reinstalling via DMG, VS Code was deleted again.

The reason is that ShipIt is an independent background process. The old VS Code had already triggered an update, and ShipIt might still be queued waiting to execute. update.mode: "none" only takes effect when the VS Code main process reads the configuration — it can't stop an already-running ShipIt.

Final Fix

1. Kill the ShipIt Process

pkill -f "ShipIt"

2. Recover the App from the Latest Temporary Directory

mv "/private/var/folders/.../T/com.microsoft.VSCode.ShipIt.xxxxxxxx/Visual Studio Code.app" \
   "/Volumes/<external-volume>/applications/"

3. Disable Auto-Updater Execution Permissions

The key is to prevent the ShipIt binary from continuing to run:

chmod -x "/Volumes/<external-volume>/applications/Visual Studio Code.app/Contents/Frameworks/Squirrel.framework/Versions/A/Resources/ShipIt"

Confirm the execute permission has been removed:

ls -la "/Volumes/<external-volume>/applications/Visual Studio Code.app/Contents/Frameworks/Squirrel.framework/Versions/A/Resources/ShipIt"

If there's no x in the permissions, it can no longer be directly executed.

4. Clean Up Temporary Update Directories

rm -rf /private/var/folders/.../T/com.microsoft.VSCode.ShipIt.*

Be careful with the path here — make sure you only delete VS Code's ShipIt temporary directories.

Fix Results

• VS Code starts normally.
• Dock icon is restored.
• Can be reopened after quitting.
• Auto-update no longer launches ShipIt.
• Future updates require manually downloading DMG or using a package manager.

Lessons Learned

update.mode: "none" Is Not Enough

It can only prevent the VS Code main process from triggering updates subsequently. It has no effect on an already-running ShipIt background process.

Installing Apps on Non-System Volumes Increases Update Risk

If you must put apps on an external volume, it's recommended to disable auto-update and switch to manual updates. Otherwise, every auto-update could trigger a cross-volume copy.

Temporary Directories Can Be Lifesavers

When an update fails, the new version of the app is often still intact at:

/private/var/folders/.../T/com.microsoft.VSCode.ShipIt.xxxxxxxx/

Don't rush to re-download — check the temporary directory first.

ShipIt Is an Independent Process

Closing VS Code doesn't mean the updater has stopped. When troubleshooting this type of issue, check and clean up ShipIt separately.

Technical Notes

| Item | Details |
|

Network Related

Benchmark Related

Prerequisites

Windows system needs WSL enabled first.

If you previously installed Kali WSL, you can uninstall the old instance with the following command:

wsl --unregister kali-linux

Procedure

1. Install Kali, create a user password as prompted, and enter the system.

wsl --install -d kali-linux

2. Package the current Kali system and save it to an intermediate location.

wsl --export kali-linux D:\kali-backup.tar

3. Delete the current Kali system.

wsl --unregister kali-linux

4. Import a new Kali system on the E drive using the package exported in step 2.

wsl --import kali-linux E:\WSL\Kali D:\kali-backup.tar

5. Log in to the Kali system.

wsl -d kali-linux

At this point you'll be logged in as root by default.

6. Create a configuration file to specify that WSL should log in with the checo user by default.

echo -e "[user]\ndefault=checo" > /etc/wsl.conf

7. Exit and restart WSL.

wsl --terminate kali-linux

wsl -d kali-linux

After re-entering, the prompt changes to $ and the username becomes checo.

Summary

This way you can put Kali WSL on a non-system drive, reducing C drive usage. If I continue writing later, I can add another article on Kali common tools and WSL network configuration.