Fall back to first visible GPU when local rank exceeds device count

[rishabhsai]

Fixes #3259

Description:

Launching with srun --ntasks-per-gpu=1 --nodes=2 --gpus-per-node=4 gives each task a single visible GPU, but SLURM_LOCALID keeps counting tasks on the node, so idist.initialize("nccl") reaches torch.cuda.set_device(3) in a process that can only see device 0 and every task with a local rank above 0 dies. ignite/distributed/comp_models/native.py now uses the first visible device when the local rank is not a valid device index, which is the fallback suggested in this issue, and device() stops warning about the mismatch in that case. auto_model passes the current CUDA device to DistributedDataParallel instead of the local rank, so device_ids stays consistent with idist.device(), which is the device the model was already moved to. This was checked with a unit test that fakes the visible device count and with the existing distributed test suites on CPU, not on a real Slurm cluster.

Check list:

• New tests are added (if a new feature is added)
• New doc strings: description and/or example code are in RST format
• Documentation is updated (if required)

[vfdev-5]

@rishabhsai can you check your PR with on SLURM and first repro the issue and then check the fix?

[rishabhsai]

Haven't been able to get on a multi-GPU machine yet, so I haven't reproduced the original failure end to end. The unit test covers the device-index logic (it fakes the visible device count and fails on main with set_device(3) instead of set_device(0)), but that's not the same as seeing it under a real allocation.

I'll follow up when I have access, or happy for you to take it from here if that's faster.

One question in the meantime: I can reproduce the same condition on a two-GPU box by setting the SLURM env vars ignite reads with one GPU visible per process, which hits the same set_device path without a scheduler. Would that be useful to you, or do you specifically want it verified under Slurm?

[vfdev-5]

The original issue is with slurm, so first we need to understand in very detail the problem with the config described in the issue and reproduce it on slurm. Next, propose a change that wont break current behavior (with and without slurm) and once approved finally implement it.
It is not the point to merge something that does not solve the original issue (even if you try to mock the slurm env vars)

[rishabhsai]

The original issue is with slurm, so first we need to understand in very detail the problem with the config described in the issue and reproduce it on slurm. Next, propose a change that wont break current behavior (with and without slurm) and once approved finally implement it.

It is not the point to merge something that does not solve the original issue (even if you try to mock the slurm env vars)

that makes sense, will do!

thanks!